Note: This is test shard 8 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from TableRangeRebalancingTest
[ RUN ] TableRangeRebalancingTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 07:59:03.740024 398 test_util.cc:268] Using random seed: 400484965
I20240503 07:59:03.746356 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:03.746404 398 ts_itest-base.cc:113] --------------
I20240503 07:59:03.746419 398 ts_itest-base.cc:114] 3 tablet servers
I20240503 07:59:03.746435 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:59:03.746448 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:03Z Disabled control of system clock
I20240503 07:59:03.754379 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:33869
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:33869
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 07:59:03.854804 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:03.854998 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:03.855036 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:03.857892 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:03.857947 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:03.857967 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:03.857983 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:03.857998 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:03.861409 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:33869
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:03.862139 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:03.863140 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:03.870110 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:03.870716 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:03.871291 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:03.871861 410 server_base.cc:1034] running on GCE node
I20240503 07:59:03.872287 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:03.873111 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:03.874250 410 hybrid_clock.cc:648] HybridClock initialized: now 1714723143874212 us; error 53 us; skew 500 ppm
I20240503 07:59:03.879160 410 webserver.cc:460] Webserver started at http://127.0.99.190:33911/ using document root <none> and password file <none>
I20240503 07:59:03.879642 410 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:03.879698 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:03.879856 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:03.881415 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2"
format_stamp: "Formatted at 2024-05-03 07:59:03 on dist-test-slave-7b05"
I20240503 07:59:03.881837 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2"
format_stamp: "Formatted at 2024-05-03 07:59:03 on dist-test-slave-7b05"
I20240503 07:59:03.884754 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:03.886703 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:03.887567 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:03.887672 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2"
format_stamp: "Formatted at 2024-05-03 07:59:03 on dist-test-slave-7b05"
I20240503 07:59:03.887742 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:03.903038 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:03.903575 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:03.903719 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:03.909683 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:33869 every 8 connection(s)
I20240503 07:59:03.909682 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:33869
I20240503 07:59:03.910571 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:03.912770 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.917213 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2: Bootstrap starting.
I20240503 07:59:03.918673 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 410
I20240503 07:59:03.918808 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:03.919505 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.920295 477 log.cc:824] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:03.921991 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2: No bootstrap required, opened a new log
I20240503 07:59:03.924669 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33869 } }
I20240503 07:59:03.924845 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.924898 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d060a22d7ae5441690b0cbf3c5aa8cd2, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.925323 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33869 } }
I20240503 07:59:03.925458 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:03.925518 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:03.925619 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.926358 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33869 } }
I20240503 07:59:03.926668 477 leader_election.cc:302] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d060a22d7ae5441690b0cbf3c5aa8cd2; no voters:
I20240503 07:59:03.926934 477 leader_election.cc:288] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:03.927260 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.927469 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [term 1 LEADER]: Becoming Leader. State: Replica: d060a22d7ae5441690b0cbf3c5aa8cd2, State: Running, Role: LEADER
I20240503 07:59:03.927712 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33869 } }
I20240503 07:59:03.927898 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:03.929122 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d060a22d7ae5441690b0cbf3c5aa8cd2. Latest consensus state: current_term: 1 leader_uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33869 } } }
I20240503 07:59:03.929245 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:03.929402 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d060a22d7ae5441690b0cbf3c5aa8cd2" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 33869 } } }
I20240503 07:59:03.929838 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2 [sys.catalog]: This master's current role is: LEADER
W20240503 07:59:03.932579 497 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:59:03.932680 497 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 07:59:03.933516 495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:03.934424 495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:03.938951 495 catalog_manager.cc:1347] Generated new cluster ID: da34452a1be74627bb8b68a10002b170
I20240503 07:59:03.938997 495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:03.965073 495 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:03.966238 495 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:03.972254 495 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d060a22d7ae5441690b0cbf3c5aa8cd2: Generated new TSK 0
I20240503 07:59:03.972985 495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:03.982936 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33869
--builtin_ntp_servers=127.0.99.148:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:04.085228 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:04.085426 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:04.085470 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:04.088390 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:04.088449 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:04.088528 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:04.092171 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33869
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:04.093015 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:04.094098 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:04.102108 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:04.102664 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:04.103281 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:04.104287 501 server_base.cc:1034] running on GCE node
I20240503 07:59:04.104652 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:04.105211 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:04.106350 501 hybrid_clock.cc:648] HybridClock initialized: now 1714723144106334 us; error 32 us; skew 500 ppm
I20240503 07:59:04.108076 501 webserver.cc:460] Webserver started at http://127.0.99.129:40681/ using document root <none> and password file <none>
I20240503 07:59:04.108585 501 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:04.108642 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:04.108800 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:04.110416 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0515af5389304285976a5e774657a904"
format_stamp: "Formatted at 2024-05-03 07:59:04 on dist-test-slave-7b05"
I20240503 07:59:04.110857 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0515af5389304285976a5e774657a904"
format_stamp: "Formatted at 2024-05-03 07:59:04 on dist-test-slave-7b05"
I20240503 07:59:04.113900 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:04.115834 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:04.116732 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:04.116827 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0515af5389304285976a5e774657a904"
format_stamp: "Formatted at 2024-05-03 07:59:04 on dist-test-slave-7b05"
I20240503 07:59:04.116900 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:04.136433 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:04.137040 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:04.137234 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:04.137874 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:04.138784 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:04.138828 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:04.138865 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:04.138892 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:04.146564 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:36959
I20240503 07:59:04.146605 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:36959 every 8 connection(s)
I20240503 07:59:04.147464 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:04.148173 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 501
I20240503 07:59:04.148254 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:04.150298 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33869
--builtin_ntp_servers=127.0.99.148:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:04.156702 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33869
I20240503 07:59:04.156883 629 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:04.157366 629 heartbeater.cc:511] Master 127.0.99.190:33869 requested a full tablet report, sending...
I20240503 07:59:04.158614 441 ts_manager.cc:194] Registered new tserver with Master: 0515af5389304285976a5e774657a904 (127.0.99.129:36959)
I20240503 07:59:04.159680 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:42165
W20240503 07:59:04.250458 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:04.250645 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:04.250698 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:04.253580 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:04.253640 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:04.253718 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:04.257145 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33869
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:04.257898 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:04.258901 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:04.266582 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:04.267383 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:04.267994 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:04.268533 632 server_base.cc:1034] running on GCE node
I20240503 07:59:04.268846 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:04.269361 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:04.270498 632 hybrid_clock.cc:648] HybridClock initialized: now 1714723144270483 us; error 29 us; skew 500 ppm
I20240503 07:59:04.272092 632 webserver.cc:460] Webserver started at http://127.0.99.130:41441/ using document root <none> and password file <none>
I20240503 07:59:04.272581 632 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:04.272635 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:04.272785 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:04.274302 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "397ae9c8460445168e898ca70b98ba79"
format_stamp: "Formatted at 2024-05-03 07:59:04 on dist-test-slave-7b05"
I20240503 07:59:04.274742 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "397ae9c8460445168e898ca70b98ba79"
format_stamp: "Formatted at 2024-05-03 07:59:04 on dist-test-slave-7b05"
I20240503 07:59:04.277668 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:04.279537 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:04.280441 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:04.280534 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "397ae9c8460445168e898ca70b98ba79"
format_stamp: "Formatted at 2024-05-03 07:59:04 on dist-test-slave-7b05"
I20240503 07:59:04.280604 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:04.295905 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:04.296433 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:04.296567 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:04.297013 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:04.297813 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:04.297857 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:04.297904 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:04.297933 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:04.306403 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:36463
I20240503 07:59:04.306455 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:36463 every 8 connection(s)
I20240503 07:59:04.310413 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:04.315060 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 632
I20240503 07:59:04.315162 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:04.317299 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33869
--builtin_ntp_servers=127.0.99.148:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:04.318745 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33869
I20240503 07:59:04.318940 760 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:04.319415 760 heartbeater.cc:511] Master 127.0.99.190:33869 requested a full tablet report, sending...
I20240503 07:59:04.320340 441 ts_manager.cc:194] Registered new tserver with Master: 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463)
I20240503 07:59:04.320968 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:44507
W20240503 07:59:04.418732 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:04.418949 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:04.419001 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:04.422040 763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:04.422108 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:04.422190 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:04.425717 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:33869
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:04.426555 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:04.427614 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:04.435532 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:04.436100 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:04.436703 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:04.437575 763 server_base.cc:1034] running on GCE node
I20240503 07:59:04.437935 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:04.438475 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:04.439620 763 hybrid_clock.cc:648] HybridClock initialized: now 1714723144439597 us; error 37 us; skew 500 ppm
I20240503 07:59:04.441406 763 webserver.cc:460] Webserver started at http://127.0.99.131:44995/ using document root <none> and password file <none>
I20240503 07:59:04.441922 763 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:04.441983 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:04.442145 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:04.443717 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ccf327c1937c416ea88ceac16c79d882"
format_stamp: "Formatted at 2024-05-03 07:59:04 on dist-test-slave-7b05"
I20240503 07:59:04.444185 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ccf327c1937c416ea88ceac16c79d882"
format_stamp: "Formatted at 2024-05-03 07:59:04 on dist-test-slave-7b05"
I20240503 07:59:04.447346 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 07:59:04.449352 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:04.450301 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:59:04.450397 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ccf327c1937c416ea88ceac16c79d882"
format_stamp: "Formatted at 2024-05-03 07:59:04 on dist-test-slave-7b05"
I20240503 07:59:04.450470 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:04.472005 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:04.472658 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:04.472800 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:04.473341 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:04.474220 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:04.474267 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:04.474316 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:04.474341 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:04.483507 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:46627
I20240503 07:59:04.483529 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:46627 every 8 connection(s)
I20240503 07:59:04.484474 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:04.492341 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 763
I20240503 07:59:04.492446 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:04.492863 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:33869
I20240503 07:59:04.493116 891 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:04.493728 891 heartbeater.cc:511] Master 127.0.99.190:33869 requested a full tablet report, sending...
I20240503 07:59:04.494670 440 ts_manager.cc:194] Registered new tserver with Master: ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:04.495236 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:52377
I20240503 07:59:04.506588 398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 07:59:04.517033 398 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240503 07:59:05.161685 629 heartbeater.cc:503] Master 127.0.99.190:33869 was elected leader, sending a full tablet report...
I20240503 07:59:05.322739 760 heartbeater.cc:503] Master 127.0.99.190:33869 was elected leader, sending a full tablet report...
I20240503 07:59:05.496946 891 heartbeater.cc:503] Master 127.0.99.190:33869 was elected leader, sending a full tablet report...
I20240503 07:59:08.269537 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43604:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\200\007\001\000\000\000\000\000\006\001\000\000\000\000\000\007\001\000\377\377\377\177"
indirect_data: "<redacted>"""
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 8
seed: 0
}
}
W20240503 07:59:08.272593 440 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 07:59:08.291236 561 tablet_service.cc:1467] Processing CreateTablet for tablet 223767c224694c59bd01b3e13ab1681c (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.291236 689 tablet_service.cc:1467] Processing CreateTablet for tablet ec7bbbf803a0485dba1adf31d79c0780 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.291457 692 tablet_service.cc:1467] Processing CreateTablet for tablet 223767c224694c59bd01b3e13ab1681c (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.292210 685 tablet_service.cc:1467] Processing CreateTablet for tablet 701ed11838a44d56a61cb1ab55394f9b (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.292429 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223767c224694c59bd01b3e13ab1681c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.292429 689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7bbbf803a0485dba1adf31d79c0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.292533 562 tablet_service.cc:1467] Processing CreateTablet for tablet 89dab389315544599d44dc6eb2a94171 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.292791 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89dab389315544599d44dc6eb2a94171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.293159 685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701ed11838a44d56a61cb1ab55394f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.293294 557 tablet_service.cc:1467] Processing CreateTablet for tablet 7dff84db44ca476e961ff8d20cbde139 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.291236 563 tablet_service.cc:1467] Processing CreateTablet for tablet 1e37fc672033429eb34645a87b617eaf (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.293530 557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dff84db44ca476e961ff8d20cbde139. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.295141 687 tablet_service.cc:1467] Processing CreateTablet for tablet a5c07a80b467401f95fd262253745d63 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.295367 687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c07a80b467401f95fd262253745d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.298277 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e37fc672033429eb34645a87b617eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.298403 555 tablet_service.cc:1467] Processing CreateTablet for tablet 4745a1d0110f401d825eab29be716761 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.298592 555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4745a1d0110f401d825eab29be716761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.300166 909 tablet_bootstrap.cc:492] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.300484 554 tablet_service.cc:1467] Processing CreateTablet for tablet 701ed11838a44d56a61cb1ab55394f9b (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.300722 554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701ed11838a44d56a61cb1ab55394f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.300805 686 tablet_service.cc:1467] Processing CreateTablet for tablet 4745a1d0110f401d825eab29be716761 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.301009 686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4745a1d0110f401d825eab29be716761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.302249 909 tablet_bootstrap.cc:654] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.302986 909 log.cc:824] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:08.303220 556 tablet_service.cc:1467] Processing CreateTablet for tablet a5c07a80b467401f95fd262253745d63 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.303439 556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c07a80b467401f95fd262253745d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.303017 825 tablet_service.cc:1467] Processing CreateTablet for tablet 1e37fc672033429eb34645a87b617eaf (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.303807 910 tablet_bootstrap.cc:492] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.304141 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e37fc672033429eb34645a87b617eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.305188 560 tablet_service.cc:1467] Processing CreateTablet for tablet c580a1b69d014ee9a40121db6becb758 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.305300 824 tablet_service.cc:1467] Processing CreateTablet for tablet 89dab389315544599d44dc6eb2a94171 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.305425 560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c580a1b69d014ee9a40121db6becb758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.305531 559 tablet_service.cc:1467] Processing CreateTablet for tablet 266c2fd4d5de4629b15db65a1d25ee17 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.305745 559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266c2fd4d5de4629b15db65a1d25ee17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.305722 822 tablet_service.cc:1467] Processing CreateTablet for tablet c580a1b69d014ee9a40121db6becb758 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.306116 821 tablet_service.cc:1467] Processing CreateTablet for tablet 266c2fd4d5de4629b15db65a1d25ee17 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.306543 820 tablet_service.cc:1467] Processing CreateTablet for tablet ec7bbbf803a0485dba1adf31d79c0780 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.306958 819 tablet_service.cc:1467] Processing CreateTablet for tablet 7dff84db44ca476e961ff8d20cbde139 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.305415 910 tablet_bootstrap.cc:654] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.307364 818 tablet_service.cc:1467] Processing CreateTablet for tablet a5c07a80b467401f95fd262253745d63 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.307456 558 tablet_service.cc:1467] Processing CreateTablet for tablet ec7bbbf803a0485dba1adf31d79c0780 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.307651 558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7bbbf803a0485dba1adf31d79c0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.307754 910 log.cc:824] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:08.307752 817 tablet_service.cc:1467] Processing CreateTablet for tablet 4745a1d0110f401d825eab29be716761 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.308168 816 tablet_service.cc:1467] Processing CreateTablet for tablet 701ed11838a44d56a61cb1ab55394f9b (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.308555 815 tablet_service.cc:1467] Processing CreateTablet for tablet 50fe2f54f5194abe9bdae29269e59c94 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.308934 814 tablet_service.cc:1467] Processing CreateTablet for tablet 50c534f4a6d44cb2bd113ff5f455687e (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.309118 909 tablet_bootstrap.cc:492] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.309259 909 ts_tablet_manager.cc:1367] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 07:59:08.309343 813 tablet_service.cc:1467] Processing CreateTablet for tablet 99b5ce50a1cb4c6cac929919fa4c20ca (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.309391 690 tablet_service.cc:1467] Processing CreateTablet for tablet 266c2fd4d5de4629b15db65a1d25ee17 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.309610 690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266c2fd4d5de4629b15db65a1d25ee17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.309743 812 tablet_service.cc:1467] Processing CreateTablet for tablet 76fffc32261042e6a9ce1091fdd9f110 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.310137 811 tablet_service.cc:1467] Processing CreateTablet for tablet 270327b4fc7e4edb80292bf26a819ad8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.310531 810 tablet_service.cc:1467] Processing CreateTablet for tablet aae666b55ef646e282a6c8b3bf732ad4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.303020 823 tablet_service.cc:1467] Processing CreateTablet for tablet 223767c224694c59bd01b3e13ab1681c (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.311390 688 tablet_service.cc:1467] Processing CreateTablet for tablet 7dff84db44ca476e961ff8d20cbde139 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.311614 688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dff84db44ca476e961ff8d20cbde139. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.311722 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89dab389315544599d44dc6eb2a94171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.312065 821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266c2fd4d5de4629b15db65a1d25ee17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.312108 909 raft_consensus.cc:348] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.312299 909 raft_consensus.cc:374] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.312350 909 raft_consensus.cc:729] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.312764 909 consensus_queue.cc:260] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.291457 693 tablet_service.cc:1467] Processing CreateTablet for tablet 89dab389315544599d44dc6eb2a94171 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.313401 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89dab389315544599d44dc6eb2a94171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.313597 909 ts_tablet_manager.cc:1398] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:08.313846 814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c534f4a6d44cb2bd113ff5f455687e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.313905 909 tablet_bootstrap.cc:492] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.291703 691 tablet_service.cc:1467] Processing CreateTablet for tablet c580a1b69d014ee9a40121db6becb758 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.315022 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c580a1b69d014ee9a40121db6becb758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.315202 909 tablet_bootstrap.cc:654] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.316309 909 tablet_bootstrap.cc:492] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.316373 909 ts_tablet_manager.cc:1367] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:08.316488 815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe2f54f5194abe9bdae29269e59c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.316673 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c580a1b69d014ee9a40121db6becb758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.316686 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223767c224694c59bd01b3e13ab1681c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.316781 813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99b5ce50a1cb4c6cac929919fa4c20ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.316994 909 raft_consensus.cc:348] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.317116 909 raft_consensus.cc:374] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.317167 909 raft_consensus.cc:729] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.317323 909 consensus_queue.cc:260] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.317659 909 ts_tablet_manager.cc:1398] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.317781 819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dff84db44ca476e961ff8d20cbde139. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.317831 909 tablet_bootstrap.cc:492] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.318636 687 tablet_service.cc:1467] Processing CreateTablet for tablet aae666b55ef646e282a6c8b3bf732ad4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.318823 687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae666b55ef646e282a6c8b3bf732ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.318826 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fffc32261042e6a9ce1091fdd9f110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.319793 818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5c07a80b467401f95fd262253745d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.319837 909 tablet_bootstrap.cc:654] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.320716 915 tablet_bootstrap.cc:492] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.320859 811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270327b4fc7e4edb80292bf26a819ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.320834 693 tablet_service.cc:1467] Processing CreateTablet for tablet 50c534f4a6d44cb2bd113ff5f455687e (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.321029 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c534f4a6d44cb2bd113ff5f455687e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.321921 816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701ed11838a44d56a61cb1ab55394f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.322225 915 tablet_bootstrap.cc:654] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.322899 915 log.cc:824] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:08.323125 817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4745a1d0110f401d825eab29be716761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.323088 691 tablet_service.cc:1467] Processing CreateTablet for tablet 50fe2f54f5194abe9bdae29269e59c94 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.323277 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe2f54f5194abe9bdae29269e59c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.324452 688 tablet_service.cc:1467] Processing CreateTablet for tablet 99b5ce50a1cb4c6cac929919fa4c20ca (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.324575 909 tablet_bootstrap.cc:492] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.324643 810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae666b55ef646e282a6c8b3bf732ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.324645 909 ts_tablet_manager.cc:1367] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:08.324833 690 tablet_service.cc:1467] Processing CreateTablet for tablet 76fffc32261042e6a9ce1091fdd9f110 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.291236 694 tablet_service.cc:1467] Processing CreateTablet for tablet 1e37fc672033429eb34645a87b617eaf (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.325042 909 raft_consensus.cc:348] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.325157 909 raft_consensus.cc:374] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.325187 909 raft_consensus.cc:729] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.325263 686 tablet_service.cc:1467] Processing CreateTablet for tablet 270327b4fc7e4edb80292bf26a819ad8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.325289 909 consensus_queue.cc:260] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.325541 909 ts_tablet_manager.cc:1398] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:08.325683 909 tablet_bootstrap.cc:492] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.325683 820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7bbbf803a0485dba1adf31d79c0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.324643 688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99b5ce50a1cb4c6cac929919fa4c20ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.326483 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e37fc672033429eb34645a87b617eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.327567 909 tablet_bootstrap.cc:654] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.327975 690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fffc32261042e6a9ce1091fdd9f110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.328253 560 tablet_service.cc:1467] Processing CreateTablet for tablet 76fffc32261042e6a9ce1091fdd9f110 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.328517 560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fffc32261042e6a9ce1091fdd9f110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.328578 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223767c224694c59bd01b3e13ab1681c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.329320 910 tablet_bootstrap.cc:492] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.329488 910 ts_tablet_manager.cc:1367] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240503 07:59:08.330528 557 tablet_service.cc:1467] Processing CreateTablet for tablet 50fe2f54f5194abe9bdae29269e59c94 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.330749 557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50fe2f54f5194abe9bdae29269e59c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.332311 686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270327b4fc7e4edb80292bf26a819ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.332975 910 raft_consensus.cc:348] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.333242 910 raft_consensus.cc:374] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.333336 910 raft_consensus.cc:729] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.333604 558 tablet_service.cc:1467] Processing CreateTablet for tablet 50c534f4a6d44cb2bd113ff5f455687e (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.333829 558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c534f4a6d44cb2bd113ff5f455687e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.333837 910 consensus_queue.cc:260] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.333964 559 tablet_service.cc:1467] Processing CreateTablet for tablet 99b5ce50a1cb4c6cac929919fa4c20ca (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.334162 559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99b5ce50a1cb4c6cac929919fa4c20ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.334362 909 tablet_bootstrap.cc:492] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.334447 909 ts_tablet_manager.cc:1367] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240503 07:59:08.334832 909 raft_consensus.cc:348] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.334923 909 raft_consensus.cc:374] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.334952 909 raft_consensus.cc:729] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.335047 909 consensus_queue.cc:260] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.335247 909 ts_tablet_manager.cc:1398] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:08.335395 909 tablet_bootstrap.cc:492] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.335704 910 ts_tablet_manager.cc:1398] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:59:08.335863 556 tablet_service.cc:1467] Processing CreateTablet for tablet 270327b4fc7e4edb80292bf26a819ad8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240503 07:59:08.336040 556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270327b4fc7e4edb80292bf26a819ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.336719 909 tablet_bootstrap.cc:654] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.336970 910 tablet_bootstrap.cc:492] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.337832 910 tablet_bootstrap.cc:654] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.337783 554 tablet_service.cc:1467] Processing CreateTablet for tablet aae666b55ef646e282a6c8b3bf732ad4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ebda6d1bfea547139a608e53decb74b9]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240503 07:59:08.338038 554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae666b55ef646e282a6c8b3bf732ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.342358 915 tablet_bootstrap.cc:492] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.342571 915 ts_tablet_manager.cc:1367] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 07:59:08.343585 909 tablet_bootstrap.cc:492] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.343681 909 ts_tablet_manager.cc:1367] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:08.343792 910 tablet_bootstrap.cc:492] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.343865 910 ts_tablet_manager.cc:1367] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:08.344036 909 raft_consensus.cc:348] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.344130 909 raft_consensus.cc:374] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.344161 909 raft_consensus.cc:729] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.344265 909 consensus_queue.cc:260] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.344372 910 raft_consensus.cc:348] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.344460 910 raft_consensus.cc:374] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.344488 910 raft_consensus.cc:729] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.344508 909 ts_tablet_manager.cc:1398] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.344594 910 consensus_queue.cc:260] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.344662 909 tablet_bootstrap.cc:492] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.344823 910 ts_tablet_manager.cc:1398] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.344980 910 tablet_bootstrap.cc:492] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.345412 909 tablet_bootstrap.cc:654] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.345815 910 tablet_bootstrap.cc:654] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.346622 915 raft_consensus.cc:348] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.346788 909 tablet_bootstrap.cc:492] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.346814 915 raft_consensus.cc:374] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.346854 909 ts_tablet_manager.cc:1367] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:08.346870 915 raft_consensus.cc:729] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.347322 909 raft_consensus.cc:348] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.347309 915 consensus_queue.cc:260] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.347420 909 raft_consensus.cc:374] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.347431 910 tablet_bootstrap.cc:492] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.347452 909 raft_consensus.cc:729] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.347494 910 ts_tablet_manager.cc:1367] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:08.347553 909 consensus_queue.cc:260] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.347785 909 ts_tablet_manager.cc:1398] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:08.347890 910 raft_consensus.cc:348] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } }
I20240503 07:59:08.347978 910 raft_consensus.cc:374] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.348006 910 raft_consensus.cc:729] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.348114 915 ts_tablet_manager.cc:1398] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:08.348104 910 consensus_queue.cc:260] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } }
I20240503 07:59:08.348331 910 ts_tablet_manager.cc:1398] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.348484 910 tablet_bootstrap.cc:492] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.348479 909 tablet_bootstrap.cc:492] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.349283 909 tablet_bootstrap.cc:654] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.349460 910 tablet_bootstrap.cc:654] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.349776 915 tablet_bootstrap.cc:492] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.350478 910 tablet_bootstrap.cc:492] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.350553 910 ts_tablet_manager.cc:1367] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:08.350950 910 raft_consensus.cc:348] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.351037 910 raft_consensus.cc:374] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.351068 910 raft_consensus.cc:729] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.351150 910 consensus_queue.cc:260] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.351378 910 ts_tablet_manager.cc:1398] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.351513 910 tablet_bootstrap.cc:492] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.351707 909 tablet_bootstrap.cc:492] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.351774 909 ts_tablet_manager.cc:1367] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:08.351841 915 tablet_bootstrap.cc:654] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.352061 909 raft_consensus.cc:348] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.352129 909 raft_consensus.cc:374] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.352149 909 raft_consensus.cc:729] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.352257 910 tablet_bootstrap.cc:654] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.352238 909 consensus_queue.cc:260] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.352463 909 ts_tablet_manager.cc:1398] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.352613 909 tablet_bootstrap.cc:492] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.352814 915 tablet_bootstrap.cc:492] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.352938 915 ts_tablet_manager.cc:1367] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:08.353229 910 tablet_bootstrap.cc:492] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.353312 910 ts_tablet_manager.cc:1367] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.353471 909 tablet_bootstrap.cc:654] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.353629 915 raft_consensus.cc:348] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.353765 915 raft_consensus.cc:374] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.353719 910 raft_consensus.cc:348] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.353812 910 raft_consensus.cc:374] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.353849 910 raft_consensus.cc:729] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.353876 915 raft_consensus.cc:729] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.353958 910 consensus_queue.cc:260] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.354187 910 ts_tablet_manager.cc:1398] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.354151 915 consensus_queue.cc:260] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.354326 910 tablet_bootstrap.cc:492] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.354404 909 tablet_bootstrap.cc:492] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.354473 909 ts_tablet_manager.cc:1367] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:08.355386 909 raft_consensus.cc:348] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.355468 909 raft_consensus.cc:374] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.355499 909 raft_consensus.cc:729] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.355531 910 tablet_bootstrap.cc:654] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.355597 909 consensus_queue.cc:260] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.355829 909 ts_tablet_manager.cc:1398] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:08.355963 909 tablet_bootstrap.cc:492] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.356029 915 ts_tablet_manager.cc:1398] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:08.356192 915 tablet_bootstrap.cc:492] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.356735 909 tablet_bootstrap.cc:654] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.356889 915 tablet_bootstrap.cc:654] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.358287 909 tablet_bootstrap.cc:492] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.358345 910 tablet_bootstrap.cc:492] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.358362 909 ts_tablet_manager.cc:1367] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:08.358414 910 ts_tablet_manager.cc:1367] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:08.358512 915 tablet_bootstrap.cc:492] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.358572 915 ts_tablet_manager.cc:1367] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:08.358748 909 raft_consensus.cc:348] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } }
I20240503 07:59:08.358829 909 raft_consensus.cc:374] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.358799 910 raft_consensus.cc:348] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.358858 909 raft_consensus.cc:729] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.358881 910 raft_consensus.cc:374] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.358911 910 raft_consensus.cc:729] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.358927 915 raft_consensus.cc:348] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.358963 909 consensus_queue.cc:260] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } }
I20240503 07:59:08.359001 915 raft_consensus.cc:374] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.359030 915 raft_consensus.cc:729] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.359038 910 consensus_queue.cc:260] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.359187 909 ts_tablet_manager.cc:1398] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.359153 915 consensus_queue.cc:260] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.359330 909 tablet_bootstrap.cc:492] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.359400 915 ts_tablet_manager.cc:1398] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:08.359405 910 ts_tablet_manager.cc:1398] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.359530 915 tablet_bootstrap.cc:492] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.359540 910 tablet_bootstrap.cc:492] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.360198 909 tablet_bootstrap.cc:654] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.360258 915 tablet_bootstrap.cc:654] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.360422 910 tablet_bootstrap.cc:654] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.361446 910 tablet_bootstrap.cc:492] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.361526 910 ts_tablet_manager.cc:1367] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.361915 910 raft_consensus.cc:348] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.362022 910 raft_consensus.cc:374] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.362061 910 raft_consensus.cc:729] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.362169 910 consensus_queue.cc:260] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.362401 910 ts_tablet_manager.cc:1398] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.362541 910 tablet_bootstrap.cc:492] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.362567 915 tablet_bootstrap.cc:492] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.362627 915 ts_tablet_manager.cc:1367] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:08.363000 915 raft_consensus.cc:348] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.363126 915 raft_consensus.cc:374] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.363191 915 raft_consensus.cc:729] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.363322 910 tablet_bootstrap.cc:654] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.363338 915 consensus_queue.cc:260] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.363615 915 ts_tablet_manager.cc:1398] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.363863 915 tablet_bootstrap.cc:492] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.364675 909 tablet_bootstrap.cc:492] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.364703 915 tablet_bootstrap.cc:654] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.364761 909 ts_tablet_manager.cc:1367] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:08.365176 909 raft_consensus.cc:348] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.365269 909 raft_consensus.cc:374] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.365303 909 raft_consensus.cc:729] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.365417 909 consensus_queue.cc:260] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.365657 909 ts_tablet_manager.cc:1398] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.365859 909 tablet_bootstrap.cc:492] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.366214 910 tablet_bootstrap.cc:492] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.366286 910 ts_tablet_manager.cc:1367] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:08.366633 909 tablet_bootstrap.cc:654] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.366636 910 raft_consensus.cc:348] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.366717 910 raft_consensus.cc:374] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.366752 910 raft_consensus.cc:729] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.366849 910 consensus_queue.cc:260] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.367074 910 ts_tablet_manager.cc:1398] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.367220 910 tablet_bootstrap.cc:492] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.367831 910 tablet_bootstrap.cc:654] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.367962 915 tablet_bootstrap.cc:492] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.368043 915 ts_tablet_manager.cc:1367] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:08.368505 915 raft_consensus.cc:348] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.368556 916 raft_consensus.cc:482] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.368645 915 raft_consensus.cc:374] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.368702 915 raft_consensus.cc:729] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.368690 916 raft_consensus.cc:504] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.368806 909 tablet_bootstrap.cc:492] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.368862 909 ts_tablet_manager.cc:1367] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:08.368820 915 consensus_queue.cc:260] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.369042 915 ts_tablet_manager.cc:1398] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.369192 915 tablet_bootstrap.cc:492] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.369225 909 raft_consensus.cc:348] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.369302 909 raft_consensus.cc:374] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.369326 909 raft_consensus.cc:729] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.369418 909 consensus_queue.cc:260] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.369637 909 ts_tablet_manager.cc:1398] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.369773 909 tablet_bootstrap.cc:492] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.369823 916 leader_election.cc:288] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.369990 915 tablet_bootstrap.cc:654] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.370509 909 tablet_bootstrap.cc:654] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.372104 910 tablet_bootstrap.cc:492] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.372177 910 ts_tablet_manager.cc:1367] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:08.372503 910 raft_consensus.cc:348] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.372577 910 raft_consensus.cc:374] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.372611 910 raft_consensus.cc:729] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.372716 910 consensus_queue.cc:260] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.372929 910 ts_tablet_manager.cc:1398] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.373111 910 tablet_bootstrap.cc:492] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.374357 910 tablet_bootstrap.cc:654] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.374540 909 tablet_bootstrap.cc:492] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.374641 909 ts_tablet_manager.cc:1367] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:08.374985 909 raft_consensus.cc:348] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.375095 909 raft_consensus.cc:374] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.375138 909 raft_consensus.cc:729] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.375335 909 consensus_queue.cc:260] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.375523 910 tablet_bootstrap.cc:492] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.375586 910 ts_tablet_manager.cc:1367] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:08.375613 909 ts_tablet_manager.cc:1398] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.375790 909 tablet_bootstrap.cc:492] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.375923 910 raft_consensus.cc:348] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.376004 910 raft_consensus.cc:374] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.376041 910 raft_consensus.cc:729] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.376164 910 consensus_queue.cc:260] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.376459 910 ts_tablet_manager.cc:1398] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.376650 910 tablet_bootstrap.cc:492] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.376982 909 tablet_bootstrap.cc:654] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.377462 910 tablet_bootstrap.cc:654] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.378438 915 tablet_bootstrap.cc:492] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.378438 910 tablet_bootstrap.cc:492] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.378497 915 ts_tablet_manager.cc:1367] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 07:59:08.378506 910 ts_tablet_manager.cc:1367] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.378885 915 raft_consensus.cc:348] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.378883 910 raft_consensus.cc:348] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.378976 915 raft_consensus.cc:374] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.379014 915 raft_consensus.cc:729] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.379128 915 consensus_queue.cc:260] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.379197 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701ed11838a44d56a61cb1ab55394f9b" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.379379 915 ts_tablet_manager.cc:1398] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:08.379426 583 raft_consensus.cc:2454] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 0.
I20240503 07:59:08.379537 915 tablet_bootstrap.cc:492] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.379552 909 tablet_bootstrap.cc:492] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.379616 909 ts_tablet_manager.cc:1367] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:08.378975 910 raft_consensus.cc:374] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.379755 910 raft_consensus.cc:729] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.379834 650 leader_election.cc:302] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, 397ae9c8460445168e898ca70b98ba79; no voters:
I20240503 07:59:08.379881 910 consensus_queue.cc:260] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.379989 909 raft_consensus.cc:348] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.380064 909 raft_consensus.cc:374] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.380082 912 raft_consensus.cc:2781] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.380149 912 raft_consensus.cc:482] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.380162 910 ts_tablet_manager.cc:1398] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.380198 912 raft_consensus.cc:3037] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.380275 910 tablet_bootstrap.cc:492] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.380403 915 tablet_bootstrap.cc:654] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.380829 910 tablet_bootstrap.cc:654] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.380091 909 raft_consensus.cc:729] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.381471 915 tablet_bootstrap.cc:492] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.381430 909 consensus_queue.cc:260] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.381536 915 ts_tablet_manager.cc:1367] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:08.381716 909 ts_tablet_manager.cc:1398] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.381878 909 tablet_bootstrap.cc:492] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.381991 915 raft_consensus.cc:348] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.382112 915 raft_consensus.cc:374] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.382143 915 raft_consensus.cc:729] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.382265 915 consensus_queue.cc:260] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.382494 915 ts_tablet_manager.cc:1398] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.382632 915 tablet_bootstrap.cc:492] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.382725 909 tablet_bootstrap.cc:654] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.383426 915 tablet_bootstrap.cc:654] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.383704 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701ed11838a44d56a61cb1ab55394f9b" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882" is_pre_election: true
W20240503 07:59:08.384348 649 leader_election.cc:341] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:08.384483 915 tablet_bootstrap.cc:492] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.384542 915 ts_tablet_manager.cc:1367] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.384894 915 raft_consensus.cc:348] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.384985 915 raft_consensus.cc:374] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.385020 915 raft_consensus.cc:729] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.385165 915 consensus_queue.cc:260] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.385310 910 tablet_bootstrap.cc:492] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.385396 910 ts_tablet_manager.cc:1367] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:08.385427 915 ts_tablet_manager.cc:1398] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.385567 915 tablet_bootstrap.cc:492] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.385617 909 tablet_bootstrap.cc:492] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.385677 909 ts_tablet_manager.cc:1367] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:08.385850 910 raft_consensus.cc:348] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.385944 910 raft_consensus.cc:374] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.385982 910 raft_consensus.cc:729] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.386018 909 raft_consensus.cc:348] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.386097 909 raft_consensus.cc:374] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.386126 909 raft_consensus.cc:729] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.386090 910 consensus_queue.cc:260] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.386286 915 tablet_bootstrap.cc:654] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.386375 910 ts_tablet_manager.cc:1398] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.386482 909 consensus_queue.cc:260] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.386801 909 ts_tablet_manager.cc:1398] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.386888 910 tablet_bootstrap.cc:492] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.387027 909 tablet_bootstrap.cc:492] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.387743 915 tablet_bootstrap.cc:492] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.387822 915 ts_tablet_manager.cc:1367] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.388074 909 tablet_bootstrap.cc:654] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.388193 910 tablet_bootstrap.cc:654] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.388177 915 raft_consensus.cc:348] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.388262 915 raft_consensus.cc:374] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.388295 915 raft_consensus.cc:729] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.388410 915 consensus_queue.cc:260] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.388631 915 ts_tablet_manager.cc:1398] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.388792 915 tablet_bootstrap.cc:492] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.388847 916 raft_consensus.cc:482] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.388903 916 raft_consensus.cc:504] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.389219 916 leader_election.cc:288] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.389613 915 tablet_bootstrap.cc:654] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.389570 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4745a1d0110f401d825eab29be716761" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882" is_pre_election: true
I20240503 07:59:08.389806 845 raft_consensus.cc:2454] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 0.
I20240503 07:59:08.390034 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4745a1d0110f401d825eab29be716761" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.390146 583 raft_consensus.cc:2454] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 0.
I20240503 07:59:08.390201 909 tablet_bootstrap.cc:492] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.390269 909 ts_tablet_manager.cc:1367] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:08.390616 915 tablet_bootstrap.cc:492] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.390621 650 leader_election.cc:302] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, 397ae9c8460445168e898ca70b98ba79; no voters:
I20240503 07:59:08.390712 915 ts_tablet_manager.cc:1367] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.390908 916 raft_consensus.cc:2781] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.390967 916 raft_consensus.cc:482] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.391005 916 raft_consensus.cc:3037] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.391096 915 raft_consensus.cc:348] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.391178 915 raft_consensus.cc:374] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.391207 915 raft_consensus.cc:729] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.390626 909 raft_consensus.cc:348] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.391247 909 raft_consensus.cc:374] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.391273 909 raft_consensus.cc:729] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.391306 915 consensus_queue.cc:260] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.391358 909 consensus_queue.cc:260] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.391527 915 ts_tablet_manager.cc:1398] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.391664 915 tablet_bootstrap.cc:492] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.391655 909 ts_tablet_manager.cc:1398] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.391786 909 tablet_bootstrap.cc:492] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79: Bootstrap starting.
I20240503 07:59:08.392570 909 tablet_bootstrap.cc:654] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.392648 915 tablet_bootstrap.cc:654] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.393260 912 raft_consensus.cc:504] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.393599 915 tablet_bootstrap.cc:492] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.393613 912 leader_election.cc:288] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.393662 915 ts_tablet_manager.cc:1367] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.393733 912 raft_consensus.cc:482] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.393787 912 raft_consensus.cc:504] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.394034 912 leader_election.cc:288] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.394037 915 raft_consensus.cc:348] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.394068 916 raft_consensus.cc:504] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.394116 915 raft_consensus.cc:374] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.394156 915 raft_consensus.cc:729] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.394258 915 consensus_queue.cc:260] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.394346 916 leader_election.cc:288] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.394496 915 ts_tablet_manager.cc:1398] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.394649 915 tablet_bootstrap.cc:492] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.394860 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89dab389315544599d44dc6eb2a94171" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882" is_pre_election: true
I20240503 07:59:08.394980 844 raft_consensus.cc:2454] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 0.
I20240503 07:59:08.395220 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4745a1d0110f401d825eab29be716761" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882"
I20240503 07:59:08.395327 843 raft_consensus.cc:3037] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.395449 915 tablet_bootstrap.cc:654] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.395563 910 tablet_bootstrap.cc:492] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.395726 910 ts_tablet_manager.cc:1367] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240503 07:59:08.396113 910 raft_consensus.cc:348] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.396198 910 raft_consensus.cc:374] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.396191 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89dab389315544599d44dc6eb2a94171" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.396260 910 raft_consensus.cc:729] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.396281 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4745a1d0110f401d825eab29be716761" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.396313 582 raft_consensus.cc:2454] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 0.
I20240503 07:59:08.396384 910 consensus_queue.cc:260] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.396415 581 raft_consensus.cc:3037] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.396615 650 leader_election.cc:302] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, 397ae9c8460445168e898ca70b98ba79; no voters:
I20240503 07:59:08.396682 910 ts_tablet_manager.cc:1398] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.396817 910 tablet_bootstrap.cc:492] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.396809 956 raft_consensus.cc:2781] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.396890 956 raft_consensus.cc:482] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.396953 956 raft_consensus.cc:3037] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.397228 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701ed11838a44d56a61cb1ab55394f9b" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882"
I20240503 07:59:08.397327 845 raft_consensus.cc:3037] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.397651 910 tablet_bootstrap.cc:654] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.397749 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701ed11838a44d56a61cb1ab55394f9b" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.397857 583 raft_consensus.cc:3037] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.398769 910 tablet_bootstrap.cc:492] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.398849 910 ts_tablet_manager.cc:1367] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:08.398962 909 tablet_bootstrap.cc:492] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79: No bootstrap required, opened a new log
I20240503 07:59:08.399032 909 ts_tablet_manager.cc:1367] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:08.399122 915 tablet_bootstrap.cc:492] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.399196 915 ts_tablet_manager.cc:1367] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:08.399220 910 raft_consensus.cc:348] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.399296 910 raft_consensus.cc:374] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.399324 910 raft_consensus.cc:729] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.399408 909 raft_consensus.cc:348] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.399425 910 consensus_queue.cc:260] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.399495 909 raft_consensus.cc:374] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.399525 909 raft_consensus.cc:729] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.399628 909 consensus_queue.cc:260] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.399677 910 ts_tablet_manager.cc:1398] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.399672 915 raft_consensus.cc:348] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.399811 910 tablet_bootstrap.cc:492] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.399832 915 raft_consensus.cc:374] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.399857 909 ts_tablet_manager.cc:1398] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.399864 915 raft_consensus.cc:729] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.399987 915 consensus_queue.cc:260] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.400239 915 ts_tablet_manager.cc:1398] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.400434 915 tablet_bootstrap.cc:492] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
W20240503 07:59:08.400748 630 tablet.cc:2369] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:08.401118 910 tablet_bootstrap.cc:654] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.401191 915 tablet_bootstrap.cc:654] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.402101 915 tablet_bootstrap.cc:492] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.402171 915 ts_tablet_manager.cc:1367] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.402549 915 raft_consensus.cc:348] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.402639 915 raft_consensus.cc:374] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.402668 915 raft_consensus.cc:729] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.402776 915 consensus_queue.cc:260] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.402988 915 ts_tablet_manager.cc:1398] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.403126 915 tablet_bootstrap.cc:492] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.403815 910 tablet_bootstrap.cc:492] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.403838 915 tablet_bootstrap.cc:654] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.403893 910 ts_tablet_manager.cc:1367] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:08.404260 910 raft_consensus.cc:348] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.404352 910 raft_consensus.cc:374] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.404460 910 raft_consensus.cc:729] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.404594 910 consensus_queue.cc:260] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.404847 915 tablet_bootstrap.cc:492] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.404865 910 ts_tablet_manager.cc:1398] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.404933 915 ts_tablet_manager.cc:1367] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.405332 910 tablet_bootstrap.cc:492] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904: Bootstrap starting.
I20240503 07:59:08.405406 915 raft_consensus.cc:348] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.405505 915 raft_consensus.cc:374] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.405548 915 raft_consensus.cc:729] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.405690 915 consensus_queue.cc:260] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.406033 915 ts_tablet_manager.cc:1398] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.406219 915 tablet_bootstrap.cc:492] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.406543 910 tablet_bootstrap.cc:654] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.407065 915 tablet_bootstrap.cc:654] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.407573 910 tablet_bootstrap.cc:492] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904: No bootstrap required, opened a new log
I20240503 07:59:08.407687 910 ts_tablet_manager.cc:1367] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.408093 910 raft_consensus.cc:348] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.408196 910 raft_consensus.cc:374] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.408239 910 raft_consensus.cc:729] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0515af5389304285976a5e774657a904, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.408248 915 tablet_bootstrap.cc:492] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.408318 915 ts_tablet_manager.cc:1367] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:08.408370 910 consensus_queue.cc:260] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.408675 910 ts_tablet_manager.cc:1398] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.408754 915 raft_consensus.cc:348] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.408852 915 raft_consensus.cc:374] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.408891 915 raft_consensus.cc:729] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.409026 915 consensus_queue.cc:260] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.409305 915 ts_tablet_manager.cc:1398] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.409462 915 tablet_bootstrap.cc:492] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882: Bootstrap starting.
I20240503 07:59:08.410121 915 tablet_bootstrap.cc:654] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.411042 915 tablet_bootstrap.cc:492] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882: No bootstrap required, opened a new log
I20240503 07:59:08.411110 915 ts_tablet_manager.cc:1367] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.411412 915 raft_consensus.cc:348] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } }
I20240503 07:59:08.411495 915 raft_consensus.cc:374] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.411525 915 raft_consensus.cc:729] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.411623 915 consensus_queue.cc:260] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } }
I20240503 07:59:08.411862 915 ts_tablet_manager.cc:1398] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.426820 920 raft_consensus.cc:482] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.426909 920 raft_consensus.cc:504] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.427781 920 leader_election.cc:288] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.431628 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c580a1b69d014ee9a40121db6becb758" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882" is_pre_election: true
I20240503 07:59:08.431742 844 raft_consensus.cc:2454] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 0.
I20240503 07:59:08.431790 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c580a1b69d014ee9a40121db6becb758" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.432020 714 raft_consensus.cc:2454] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 0.
I20240503 07:59:08.432075 518 leader_election.cc:302] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.432304 920 raft_consensus.cc:2781] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.432366 920 raft_consensus.cc:482] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.432410 920 raft_consensus.cc:3037] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.440629 843 raft_consensus.cc:2454] T 4745a1d0110f401d825eab29be716761 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.440632 581 raft_consensus.cc:2454] T 4745a1d0110f401d825eab29be716761 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.440639 956 raft_consensus.cc:504] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.440925 650 leader_election.cc:302] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, 397ae9c8460445168e898ca70b98ba79; no voters:
I20240503 07:59:08.440984 956 leader_election.cc:288] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.441077 956 raft_consensus.cc:2781] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.441150 956 raft_consensus.cc:686] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [term 1 LEADER]: Becoming Leader. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Running, Role: LEADER
I20240503 07:59:08.441421 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89dab389315544599d44dc6eb2a94171" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.441443 956 consensus_queue.cc:237] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.441516 581 raft_consensus.cc:3037] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.441610 845 raft_consensus.cc:2454] T 701ed11838a44d56a61cb1ab55394f9b P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.442286 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89dab389315544599d44dc6eb2a94171" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882"
I20240503 07:59:08.442345 581 raft_consensus.cc:2454] T 89dab389315544599d44dc6eb2a94171 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.442381 845 raft_consensus.cc:3037] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.442616 650 leader_election.cc:302] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, 397ae9c8460445168e898ca70b98ba79; no voters:
I20240503 07:59:08.442760 957 raft_consensus.cc:2781] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.442821 957 raft_consensus.cc:686] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [term 1 LEADER]: Becoming Leader. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Running, Role: LEADER
I20240503 07:59:08.442921 957 consensus_queue.cc:237] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.443096 845 raft_consensus.cc:2454] T 89dab389315544599d44dc6eb2a94171 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.443207 649 leader_election.cc:302] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.443362 912 raft_consensus.cc:2781] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.443470 912 raft_consensus.cc:686] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [term 1 LEADER]: Becoming Leader. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Running, Role: LEADER
I20240503 07:59:08.443567 912 consensus_queue.cc:237] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.444850 583 raft_consensus.cc:2454] T 701ed11838a44d56a61cb1ab55394f9b P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.446002 439 catalog_manager.cc:5484] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 397ae9c8460445168e898ca70b98ba79 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "397ae9c8460445168e898ca70b98ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.446590 439 catalog_manager.cc:5484] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 397ae9c8460445168e898ca70b98ba79 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "397ae9c8460445168e898ca70b98ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.447726 925 raft_consensus.cc:482] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.447821 925 raft_consensus.cc:504] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.448020 973 raft_consensus.cc:482] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.448091 973 raft_consensus.cc:504] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.448724 925 leader_election.cc:288] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), 0515af5389304285976a5e774657a904 (127.0.99.129:36959)
I20240503 07:59:08.449615 973 leader_election.cc:288] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463)
I20240503 07:59:08.449921 439 catalog_manager.cc:5484] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 397ae9c8460445168e898ca70b98ba79 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "397ae9c8460445168e898ca70b98ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.452705 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266c2fd4d5de4629b15db65a1d25ee17" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.452839 583 raft_consensus.cc:2454] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.453131 781 leader_election.cc:302] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.453162 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe2f54f5194abe9bdae29269e59c94" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.453171 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266c2fd4d5de4629b15db65a1d25ee17" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.453263 714 raft_consensus.cc:2454] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.453286 713 raft_consensus.cc:2454] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.452724 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe2f54f5194abe9bdae29269e59c94" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.453378 973 raft_consensus.cc:2781] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.453435 973 raft_consensus.cc:482] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.453420 581 raft_consensus.cc:2454] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.453473 973 raft_consensus.cc:3037] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.453651 780 leader_election.cc:302] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.454201 925 raft_consensus.cc:2781] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.454250 925 raft_consensus.cc:482] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.454274 925 raft_consensus.cc:3037] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.460523 976 raft_consensus.cc:482] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.460592 976 raft_consensus.cc:504] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.460875 976 leader_election.cc:288] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463)
I20240503 07:59:08.461138 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b5ce50a1cb4c6cac929919fa4c20ca" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.461191 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b5ce50a1cb4c6cac929919fa4c20ca" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.461230 581 raft_consensus.cc:2454] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.461295 713 raft_consensus.cc:2454] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.461465 781 leader_election.cc:302] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.461614 976 raft_consensus.cc:2781] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.461664 976 raft_consensus.cc:482] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.461688 976 raft_consensus.cc:3037] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.467882 977 raft_consensus.cc:482] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.467952 977 raft_consensus.cc:504] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.468218 977 leader_election.cc:288] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), 0515af5389304285976a5e774657a904 (127.0.99.129:36959)
I20240503 07:59:08.468489 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7bbbf803a0485dba1adf31d79c0780" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.468484 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7bbbf803a0485dba1adf31d79c0780" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.468590 713 raft_consensus.cc:2454] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.468590 581 raft_consensus.cc:2454] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.468812 781 leader_election.cc:302] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.468967 977 raft_consensus.cc:2781] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.469024 977 raft_consensus.cc:482] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.469089 977 raft_consensus.cc:3037] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:08.487797 892 tablet.cc:2369] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:08.503307 978 raft_consensus.cc:482] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.503378 978 raft_consensus.cc:504] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.503659 978 leader_election.cc:288] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), 0515af5389304285976a5e774657a904 (127.0.99.129:36959)
I20240503 07:59:08.503940 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dff84db44ca476e961ff8d20cbde139" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.503950 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dff84db44ca476e961ff8d20cbde139" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.504042 581 raft_consensus.cc:2454] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.504045 713 raft_consensus.cc:2454] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.504295 781 leader_election.cc:302] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.504463 978 raft_consensus.cc:2781] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.504511 978 raft_consensus.cc:482] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.504570 978 raft_consensus.cc:3037] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.511499 957 raft_consensus.cc:482] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.511577 957 raft_consensus.cc:504] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.511871 957 leader_election.cc:288] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.512147 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c534f4a6d44cb2bd113ff5f455687e" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882" is_pre_election: true
I20240503 07:59:08.512147 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c534f4a6d44cb2bd113ff5f455687e" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.512259 581 raft_consensus.cc:2454] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 0.
I20240503 07:59:08.512259 845 raft_consensus.cc:2454] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 0.
I20240503 07:59:08.512480 649 leader_election.cc:302] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.512634 957 raft_consensus.cc:2781] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.512701 957 raft_consensus.cc:482] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.512735 957 raft_consensus.cc:3037] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.518760 979 raft_consensus.cc:482] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.518826 979 raft_consensus.cc:504] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } }
I20240503 07:59:08.519078 979 leader_election.cc:288] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), 0515af5389304285976a5e774657a904 (127.0.99.129:36959)
I20240503 07:59:08.519335 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e37fc672033429eb34645a87b617eaf" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.519336 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e37fc672033429eb34645a87b617eaf" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.519428 713 raft_consensus.cc:2454] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.519426 581 raft_consensus.cc:2454] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.519654 780 leader_election.cc:302] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.519802 979 raft_consensus.cc:2781] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.519860 979 raft_consensus.cc:482] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.519886 979 raft_consensus.cc:3037] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.533713 973 raft_consensus.cc:504] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.533731 957 raft_consensus.cc:504] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.533731 920 raft_consensus.cc:504] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.534025 973 leader_election.cc:288] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Requested vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463)
I20240503 07:59:08.534104 957 leader_election.cc:288] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.534092 920 leader_election.cc:288] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 election: Requested vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.534741 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c580a1b69d014ee9a40121db6becb758" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.534842 713 raft_consensus.cc:3037] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.535074 925 raft_consensus.cc:504] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.535171 976 raft_consensus.cc:504] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.535332 925 leader_election.cc:288] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Requested vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), 0515af5389304285976a5e774657a904 (127.0.99.129:36959)
I20240503 07:59:08.535498 976 leader_election.cc:288] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Requested vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463)
I20240503 07:59:08.535779 713 raft_consensus.cc:2454] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 1.
I20240503 07:59:08.536015 977 raft_consensus.cc:504] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.536065 518 leader_election.cc:302] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, 397ae9c8460445168e898ca70b98ba79; no voters:
I20240503 07:59:08.536283 977 leader_election.cc:288] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Requested vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), 0515af5389304285976a5e774657a904 (127.0.99.129:36959)
I20240503 07:59:08.536268 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c534f4a6d44cb2bd113ff5f455687e" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.536374 581 raft_consensus.cc:3037] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.536605 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266c2fd4d5de4629b15db65a1d25ee17" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.536713 583 raft_consensus.cc:3037] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.537117 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c580a1b69d014ee9a40121db6becb758" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882"
I20240503 07:59:08.537214 845 raft_consensus.cc:3037] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.537242 579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7bbbf803a0485dba1adf31d79c0780" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.537338 579 raft_consensus.cc:3037] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.537647 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe2f54f5194abe9bdae29269e59c94" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.537690 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c534f4a6d44cb2bd113ff5f455687e" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882"
I20240503 07:59:08.537745 582 raft_consensus.cc:3037] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.537775 843 raft_consensus.cc:3037] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.537878 711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7bbbf803a0485dba1adf31d79c0780" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.537963 711 raft_consensus.cc:3037] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.537951 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b5ce50a1cb4c6cac929919fa4c20ca" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.537591 580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b5ce50a1cb4c6cac929919fa4c20ca" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.537976 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266c2fd4d5de4629b15db65a1d25ee17" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.538089 580 raft_consensus.cc:3037] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.538058 712 raft_consensus.cc:3037] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.538136 713 raft_consensus.cc:3037] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.537838 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50fe2f54f5194abe9bdae29269e59c94" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.538226 714 raft_consensus.cc:3037] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.536276 920 raft_consensus.cc:2781] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.538553 920 raft_consensus.cc:686] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [term 1 LEADER]: Becoming Leader. State: Replica: 0515af5389304285976a5e774657a904, State: Running, Role: LEADER
I20240503 07:59:08.538897 920 consensus_queue.cc:237] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.541882 439 catalog_manager.cc:5484] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0515af5389304285976a5e774657a904 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "0515af5389304285976a5e774657a904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.542829 977 raft_consensus.cc:482] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.542898 977 raft_consensus.cc:504] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.543154 977 leader_election.cc:288] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463)
I20240503 07:59:08.543474 710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae666b55ef646e282a6c8b3bf732ad4" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.543496 578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae666b55ef646e282a6c8b3bf732ad4" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.543576 710 raft_consensus.cc:2454] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.543592 578 raft_consensus.cc:2454] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.543814 781 leader_election.cc:302] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.543970 977 raft_consensus.cc:2781] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.544028 977 raft_consensus.cc:482] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.544072 977 raft_consensus.cc:3037] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.554031 957 raft_consensus.cc:482] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.554105 957 raft_consensus.cc:504] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.554390 957 leader_election.cc:288] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.554664 578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fffc32261042e6a9ce1091fdd9f110" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.554675 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fffc32261042e6a9ce1091fdd9f110" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882" is_pre_election: true
I20240503 07:59:08.554755 578 raft_consensus.cc:2454] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 0.
I20240503 07:59:08.554760 844 raft_consensus.cc:2454] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 0.
I20240503 07:59:08.554975 649 leader_election.cc:302] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.555145 957 raft_consensus.cc:2781] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.555203 957 raft_consensus.cc:482] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.555240 957 raft_consensus.cc:3037] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:08.563922 761 tablet.cc:2369] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:08.575632 713 raft_consensus.cc:2454] T 266c2fd4d5de4629b15db65a1d25ee17 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.575632 581 raft_consensus.cc:2454] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.575624 978 raft_consensus.cc:504] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.575899 583 raft_consensus.cc:2454] T 266c2fd4d5de4629b15db65a1d25ee17 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.575910 978 leader_election.cc:288] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Requested vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), 0515af5389304285976a5e774657a904 (127.0.99.129:36959)
I20240503 07:59:08.575969 711 raft_consensus.cc:2454] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.576184 712 raft_consensus.cc:2454] T 99b5ce50a1cb4c6cac929919fa4c20ca P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.576386 714 raft_consensus.cc:2454] T 50fe2f54f5194abe9bdae29269e59c94 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.576691 845 raft_consensus.cc:2454] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 1.
I20240503 07:59:08.576774 843 raft_consensus.cc:2454] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.576841 979 raft_consensus.cc:504] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } }
I20240503 07:59:08.577102 649 leader_election.cc:302] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.577121 979 leader_election.cc:288] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Requested vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), 0515af5389304285976a5e774657a904 (127.0.99.129:36959)
I20240503 07:59:08.577132 957 raft_consensus.cc:504] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.577246 916 raft_consensus.cc:2781] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.577306 916 raft_consensus.cc:686] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [term 1 LEADER]: Becoming Leader. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Running, Role: LEADER
I20240503 07:59:08.577350 781 leader_election.cc:302] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.577414 916 consensus_queue.cc:237] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.577687 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fffc32261042e6a9ce1091fdd9f110" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882"
I20240503 07:59:08.577770 843 raft_consensus.cc:3037] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.577950 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fffc32261042e6a9ce1091fdd9f110" candidate_uuid: "397ae9c8460445168e898ca70b98ba79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.578034 583 raft_consensus.cc:3037] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.578439 843 raft_consensus.cc:2454] T 76fffc32261042e6a9ce1091fdd9f110 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.578681 649 leader_election.cc:302] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.578786 583 raft_consensus.cc:2454] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397ae9c8460445168e898ca70b98ba79 in term 1.
I20240503 07:59:08.578799 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dff84db44ca476e961ff8d20cbde139" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.578835 912 raft_consensus.cc:2781] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.578893 581 raft_consensus.cc:3037] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.578894 912 raft_consensus.cc:686] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [term 1 LEADER]: Becoming Leader. State: Replica: 397ae9c8460445168e898ca70b98ba79, State: Running, Role: LEADER
I20240503 07:59:08.578995 912 consensus_queue.cc:237] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.579268 979 raft_consensus.cc:2781] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.579347 979 raft_consensus.cc:686] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [term 1 LEADER]: Becoming Leader. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Running, Role: LEADER
I20240503 07:59:08.579638 581 raft_consensus.cc:2454] T 7dff84db44ca476e961ff8d20cbde139 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.579650 979 consensus_queue.cc:237] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.579972 439 catalog_manager.cc:5484] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 397ae9c8460445168e898ca70b98ba79 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "397ae9c8460445168e898ca70b98ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.577416 957 leader_election.cc:288] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [CANDIDATE]: Term 1 election: Requested vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.581138 578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e37fc672033429eb34645a87b617eaf" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.581152 781 leader_election.cc:302] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.581223 578 raft_consensus.cc:3037] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.581269 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dff84db44ca476e961ff8d20cbde139" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.581328 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e37fc672033429eb34645a87b617eaf" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.581357 714 raft_consensus.cc:3037] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.581393 712 raft_consensus.cc:3037] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.581427 978 raft_consensus.cc:2781] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.581508 978 raft_consensus.cc:686] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [term 1 LEADER]: Becoming Leader. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Running, Role: LEADER
I20240503 07:59:08.582175 780 leader_election.cc:302] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.582345 780 leader_election.cc:302] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.582475 780 leader_election.cc:302] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.582592 925 raft_consensus.cc:2781] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.582695 976 raft_consensus.cc:2781] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.582732 925 raft_consensus.cc:686] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [term 1 LEADER]: Becoming Leader. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Running, Role: LEADER
I20240503 07:59:08.582829 925 consensus_queue.cc:237] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.582861 976 raft_consensus.cc:686] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [term 1 LEADER]: Becoming Leader. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Running, Role: LEADER
I20240503 07:59:08.582995 976 consensus_queue.cc:237] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.583106 979 raft_consensus.cc:2781] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.583170 979 raft_consensus.cc:686] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [term 1 LEADER]: Becoming Leader. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Running, Role: LEADER
I20240503 07:59:08.583308 979 consensus_queue.cc:237] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.582165 978 consensus_queue.cc:237] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.583884 439 catalog_manager.cc:5484] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf327c1937c416ea88ceac16c79d882 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ccf327c1937c416ea88ceac16c79d882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:08.584388 441 catalog_manager.cc:5484] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 397ae9c8460445168e898ca70b98ba79 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "397ae9c8460445168e898ca70b98ba79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.587033 441 catalog_manager.cc:5484] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf327c1937c416ea88ceac16c79d882 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ccf327c1937c416ea88ceac16c79d882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:08.587217 441 catalog_manager.cc:5484] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf327c1937c416ea88ceac16c79d882 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ccf327c1937c416ea88ceac16c79d882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:08.587380 441 catalog_manager.cc:5484] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf327c1937c416ea88ceac16c79d882 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ccf327c1937c416ea88ceac16c79d882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:08.587532 441 catalog_manager.cc:5484] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf327c1937c416ea88ceac16c79d882 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ccf327c1937c416ea88ceac16c79d882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:08.588574 920 raft_consensus.cc:482] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.588661 920 raft_consensus.cc:504] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.588714 985 raft_consensus.cc:482] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.588789 985 raft_consensus.cc:504] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.588958 920 leader_election.cc:288] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.589352 711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270327b4fc7e4edb80292bf26a819ad8" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.589282 985 leader_election.cc:288] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463)
I20240503 07:59:08.589459 711 raft_consensus.cc:2454] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 0.
I20240503 07:59:08.589435 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223767c224694c59bd01b3e13ab1681c" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904" is_pre_election: true
I20240503 07:59:08.589540 581 raft_consensus.cc:2454] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.589551 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270327b4fc7e4edb80292bf26a819ad8" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882" is_pre_election: true
I20240503 07:59:08.589639 843 raft_consensus.cc:2454] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 0.
I20240503 07:59:08.589823 711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223767c224694c59bd01b3e13ab1681c" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.589840 781 leader_election.cc:302] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.589864 518 leader_election.cc:302] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, 397ae9c8460445168e898ca70b98ba79; no voters:
I20240503 07:59:08.589929 711 raft_consensus.cc:2454] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 0.
I20240503 07:59:08.589977 985 raft_consensus.cc:2781] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.590049 920 raft_consensus.cc:2781] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.590103 920 raft_consensus.cc:482] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.590147 920 raft_consensus.cc:3037] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.590175 985 raft_consensus.cc:482] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.590232 985 raft_consensus.cc:3037] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.612803 980 raft_consensus.cc:482] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.612897 980 raft_consensus.cc:504] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.613261 980 leader_election.cc:288] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.613526 711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c07a80b467401f95fd262253745d63" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79" is_pre_election: true
I20240503 07:59:08.613597 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c07a80b467401f95fd262253745d63" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882" is_pre_election: true
I20240503 07:59:08.613623 711 raft_consensus.cc:2454] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 0.
I20240503 07:59:08.613713 843 raft_consensus.cc:2454] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 0.
I20240503 07:59:08.613860 518 leader_election.cc:302] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, 397ae9c8460445168e898ca70b98ba79; no voters:
I20240503 07:59:08.614006 980 raft_consensus.cc:2781] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.614058 980 raft_consensus.cc:482] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.614099 980 raft_consensus.cc:3037] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.665829 579 raft_consensus.cc:2454] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.665829 712 raft_consensus.cc:2454] T 1e37fc672033429eb34645a87b617eaf P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.665836 977 raft_consensus.cc:504] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.666152 714 raft_consensus.cc:2454] T 7dff84db44ca476e961ff8d20cbde139 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.666258 977 leader_election.cc:288] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Requested vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463)
I20240503 07:59:08.666693 582 raft_consensus.cc:2454] T 50fe2f54f5194abe9bdae29269e59c94 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.666891 580 raft_consensus.cc:2454] T 99b5ce50a1cb4c6cac929919fa4c20ca P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.667097 985 raft_consensus.cc:504] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.667387 985 leader_election.cc:288] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Requested vote from peers 0515af5389304285976a5e774657a904 (127.0.99.129:36959), 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463)
I20240503 07:59:08.667524 578 raft_consensus.cc:2454] T 1e37fc672033429eb34645a87b617eaf P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.667582 920 raft_consensus.cc:504] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.667649 780 leader_election.cc:302] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397ae9c8460445168e898ca70b98ba79, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.667889 920 leader_election.cc:288] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 election: Requested vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.667902 985 raft_consensus.cc:2781] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.667964 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae666b55ef646e282a6c8b3bf732ad4" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.668022 985 raft_consensus.cc:686] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [term 1 LEADER]: Becoming Leader. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Running, Role: LEADER
I20240503 07:59:08.668061 714 raft_consensus.cc:3037] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.668133 985 consensus_queue.cc:237] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } }
I20240503 07:59:08.668584 980 raft_consensus.cc:504] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.668836 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223767c224694c59bd01b3e13ab1681c" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.668876 980 leader_election.cc:288] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 election: Requested vote from peers 397ae9c8460445168e898ca70b98ba79 (127.0.99.130:36463), ccf327c1937c416ea88ceac16c79d882 (127.0.99.131:46627)
I20240503 07:59:08.668956 712 raft_consensus.cc:3037] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.669572 578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae666b55ef646e282a6c8b3bf732ad4" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.669576 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c07a80b467401f95fd262253745d63" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.669669 713 raft_consensus.cc:3037] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.669659 580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223767c224694c59bd01b3e13ab1681c" candidate_uuid: "ccf327c1937c416ea88ceac16c79d882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0515af5389304285976a5e774657a904"
I20240503 07:59:08.669679 578 raft_consensus.cc:3037] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.669731 580 raft_consensus.cc:3037] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.669764 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270327b4fc7e4edb80292bf26a819ad8" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882"
I20240503 07:59:08.669847 843 raft_consensus.cc:3037] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.669934 711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270327b4fc7e4edb80292bf26a819ad8" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "397ae9c8460445168e898ca70b98ba79"
I20240503 07:59:08.670009 711 raft_consensus.cc:3037] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.669847 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5c07a80b467401f95fd262253745d63" candidate_uuid: "0515af5389304285976a5e774657a904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf327c1937c416ea88ceac16c79d882"
I20240503 07:59:08.670195 845 raft_consensus.cc:3037] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.671065 441 catalog_manager.cc:5484] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf327c1937c416ea88ceac16c79d882 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ccf327c1937c416ea88ceac16c79d882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.756134 843 raft_consensus.cc:2454] T 270327b4fc7e4edb80292bf26a819ad8 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 1.
I20240503 07:59:08.756142 580 raft_consensus.cc:2454] T 223767c224694c59bd01b3e13ab1681c P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.756142 714 raft_consensus.cc:2454] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.756494 578 raft_consensus.cc:2454] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.756754 781 leader_election.cc:302] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.756963 781 leader_election.cc:302] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.757138 977 raft_consensus.cc:2781] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.757222 977 raft_consensus.cc:686] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [term 1 LEADER]: Becoming Leader. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Running, Role: LEADER
I20240503 07:59:08.757266 845 raft_consensus.cc:2454] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 1.
I20240503 07:59:08.757350 977 consensus_queue.cc:237] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.757475 978 raft_consensus.cc:2781] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.757550 978 raft_consensus.cc:686] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [term 1 LEADER]: Becoming Leader. State: Replica: ccf327c1937c416ea88ceac16c79d882, State: Running, Role: LEADER
I20240503 07:59:08.757678 978 consensus_queue.cc:237] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.758112 712 raft_consensus.cc:2454] T 223767c224694c59bd01b3e13ab1681c P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf327c1937c416ea88ceac16c79d882 in term 1.
I20240503 07:59:08.758270 518 leader_election.cc:302] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.758435 518 leader_election.cc:302] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0515af5389304285976a5e774657a904, ccf327c1937c416ea88ceac16c79d882; no voters:
I20240503 07:59:08.758580 920 raft_consensus.cc:2781] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.758738 920 raft_consensus.cc:686] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [term 1 LEADER]: Becoming Leader. State: Replica: 0515af5389304285976a5e774657a904, State: Running, Role: LEADER
I20240503 07:59:08.758863 920 consensus_queue.cc:237] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.758992 711 raft_consensus.cc:2454] T 270327b4fc7e4edb80292bf26a819ad8 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 1.
I20240503 07:59:08.759043 980 raft_consensus.cc:2781] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.759204 980 raft_consensus.cc:686] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [term 1 LEADER]: Becoming Leader. State: Replica: 0515af5389304285976a5e774657a904, State: Running, Role: LEADER
I20240503 07:59:08.759315 980 consensus_queue.cc:237] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } }
I20240503 07:59:08.759510 713 raft_consensus.cc:2454] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0515af5389304285976a5e774657a904 in term 1.
I20240503 07:59:08.761114 441 catalog_manager.cc:5484] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf327c1937c416ea88ceac16c79d882 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ccf327c1937c416ea88ceac16c79d882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:08.761336 441 catalog_manager.cc:5484] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf327c1937c416ea88ceac16c79d882 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ccf327c1937c416ea88ceac16c79d882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:08.761358 439 catalog_manager.cc:5484] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0515af5389304285976a5e774657a904 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "0515af5389304285976a5e774657a904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.761556 439 catalog_manager.cc:5484] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0515af5389304285976a5e774657a904 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "0515af5389304285976a5e774657a904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:08.879266 998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:08.879467 998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:08.882385 998 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:08.889317 957 consensus_queue.cc:1035] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.893221 912 consensus_queue.cc:1035] T 89dab389315544599d44dc6eb2a94171 P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.896687 957 consensus_queue.cc:1035] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.899366 957 consensus_queue.cc:1035] T 4745a1d0110f401d825eab29be716761 P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 07:59:08.938573 957 consensus_queue.cc:1035] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.956205 912 consensus_queue.cc:1035] T 701ed11838a44d56a61cb1ab55394f9b P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.983088 713 raft_consensus.cc:1261] T a5c07a80b467401f95fd262253745d63 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Refusing update from remote peer 0515af5389304285976a5e774657a904: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.983748 994 consensus_queue.cc:1035] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.984897 845 raft_consensus.cc:1261] T a5c07a80b467401f95fd262253745d63 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Refusing update from remote peer 0515af5389304285976a5e774657a904: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.985347 994 consensus_queue.cc:1035] T a5c07a80b467401f95fd262253745d63 P 0515af5389304285976a5e774657a904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.989116 845 raft_consensus.cc:1261] T c580a1b69d014ee9a40121db6becb758 P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Refusing update from remote peer 0515af5389304285976a5e774657a904: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.990994 980 consensus_queue.cc:1035] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.993819 711 raft_consensus.cc:1261] T c580a1b69d014ee9a40121db6becb758 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Refusing update from remote peer 0515af5389304285976a5e774657a904: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.997397 916 consensus_queue.cc:1035] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.999557 711 raft_consensus.cc:1261] T ec7bbbf803a0485dba1adf31d79c0780 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Refusing update from remote peer ccf327c1937c416ea88ceac16c79d882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.000026 980 consensus_queue.cc:1035] T c580a1b69d014ee9a40121db6becb758 P 0515af5389304285976a5e774657a904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.000591 978 consensus_queue.cc:1035] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.002069 711 raft_consensus.cc:1261] T aae666b55ef646e282a6c8b3bf732ad4 P 397ae9c8460445168e898ca70b98ba79 [term 1 FOLLOWER]: Refusing update from remote peer ccf327c1937c416ea88ceac16c79d882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.002805 845 raft_consensus.cc:1261] T 50c534f4a6d44cb2bd113ff5f455687e P ccf327c1937c416ea88ceac16c79d882 [term 1 FOLLOWER]: Refusing update from remote peer 397ae9c8460445168e898ca70b98ba79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.002938 580 raft_consensus.cc:1261] T aae666b55ef646e282a6c8b3bf732ad4 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Refusing update from remote peer ccf327c1937c416ea88ceac16c79d882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.003057 578 raft_consensus.cc:1261] T ec7bbbf803a0485dba1adf31d79c0780 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Refusing update from remote peer ccf327c1937c416ea88ceac16c79d882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.003283 978 consensus_queue.cc:1035] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.003835 978 consensus_queue.cc:1035] T aae666b55ef646e282a6c8b3bf732ad4 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.004002 580 raft_consensus.cc:1261] T 50c534f4a6d44cb2bd113ff5f455687e P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Refusing update from remote peer 397ae9c8460445168e898ca70b98ba79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.004248 978 consensus_queue.cc:1035] T ec7bbbf803a0485dba1adf31d79c0780 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.004521 916 consensus_queue.cc:1035] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.004778 916 consensus_queue.cc:1035] T 50c534f4a6d44cb2bd113ff5f455687e P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.017002 1037 mvcc.cc:205] Tried to move back new op lower bound from 7023506018227478528 to 7023506017319579648. Current Snapshot: MvccSnapshot[applied={T|T < 7023506018227478528}]
I20240503 07:59:09.020098 1040 mvcc.cc:205] Tried to move back new op lower bound from 7023506018227478528 to 7023506017319579648. Current Snapshot: MvccSnapshot[applied={T|T < 7023506018227478528}]
I20240503 07:59:09.021350 583 raft_consensus.cc:1261] T 76fffc32261042e6a9ce1091fdd9f110 P 0515af5389304285976a5e774657a904 [term 1 FOLLOWER]: Refusing update from remote peer 397ae9c8460445168e898ca70b98ba79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.021853 1013 mvcc.cc:205] Tried to move back new op lower bound from 7023506018296393728 to 7023506016596914176. Current Snapshot: MvccSnapshot[applied={T|T < 7023506018296393728}]
I20240503 07:59:09.024509 957 consensus_queue.cc:1035] T 76fffc32261042e6a9ce1091fdd9f110 P 397ae9c8460445168e898ca70b98ba79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.042586 978 consensus_queue.cc:1035] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.057871 978 consensus_queue.cc:1035] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.069480 985 consensus_queue.cc:1035] T 7dff84db44ca476e961ff8d20cbde139 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 154.464 ms
time per row: 0.0772319 ms
I20240503 07:59:09.095201 979 consensus_queue.cc:1035] T 99b5ce50a1cb4c6cac929919fa4c20ca P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.129860 925 consensus_queue.cc:1035] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.133033 977 consensus_queue.cc:1035] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.133780 925 consensus_queue.cc:1035] T 266c2fd4d5de4629b15db65a1d25ee17 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.136835 925 consensus_queue.cc:1035] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.138979 979 consensus_queue.cc:1035] T 50fe2f54f5194abe9bdae29269e59c94 P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.141013 985 consensus_queue.cc:1035] T 223767c224694c59bd01b3e13ab1681c P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.223737 979 consensus_queue.cc:1035] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0515af5389304285976a5e774657a904" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.227908 979 consensus_queue.cc:1035] T 1e37fc672033429eb34645a87b617eaf P ccf327c1937c416ea88ceac16c79d882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.273020 554 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:09.273121 995 consensus_queue.cc:1035] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf327c1937c416ea88ceac16c79d882" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 46627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.273875 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:09.276671 920 consensus_queue.cc:1035] T 270327b4fc7e4edb80292bf26a819ad8 P 0515af5389304285976a5e774657a904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397ae9c8460445168e898ca70b98ba79" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.278209 825 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:09.370218 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:09.370378 554 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:09.371515 825 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:09.471385 554 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:09.472486 825 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:09.473713 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:09.704738 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:09.704762 554 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:09.705024 825 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:09.792783 554 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:09.792868 825 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:09.793388 689 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:09.824818 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 501
I20240503 07:59:09.832216 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 632
I20240503 07:59:09.837262 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 763
I20240503 07:59:09.841948 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 410
2024-05-03T07:59:09Z chronyd exiting
[ OK ] TableRangeRebalancingTest.Basic (6128 ms)
[----------] 1 test from TableRangeRebalancingTest (6129 ms total)
[----------] 1 test from RebalanceStartSafetyTest
[ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1
I20240503 07:59:09.867846 398 test_util.cc:268] Using random seed: 406612801
I20240503 07:59:09.868531 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:09.868570 398 ts_itest-base.cc:113] --------------
I20240503 07:59:09.868580 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 07:59:09.868589 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:59:09.868610 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:09Z Disabled control of system clock
I20240503 07:59:09.875437 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35123
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:35123
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 07:59:09.977283 1251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:09.977499 1251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:09.977560 1251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:09.980432 1251 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:09.980499 1251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:09.980525 1251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:09.980563 1251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:09.980595 1251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:09.984074 1251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:35123
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:09.984921 1251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:09.985987 1251 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:09.993655 1259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:09.994143 1256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:09.994997 1251 server_base.cc:1034] running on GCE node
W20240503 07:59:09.995460 1257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:09.995937 1251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:09.996788 1251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:09.997926 1251 hybrid_clock.cc:648] HybridClock initialized: now 1714723149997913 us; error 28 us; skew 500 ppm
I20240503 07:59:09.999729 1251 webserver.cc:460] Webserver started at http://127.0.99.190:39789/ using document root <none> and password file <none>
I20240503 07:59:10.000267 1251 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:10.000319 1251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:10.000492 1251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:10.002095 1251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "979ac07da07f4df9a3a3e9039a9faff8"
format_stamp: "Formatted at 2024-05-03 07:59:09 on dist-test-slave-7b05"
I20240503 07:59:10.002534 1251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "979ac07da07f4df9a3a3e9039a9faff8"
format_stamp: "Formatted at 2024-05-03 07:59:09 on dist-test-slave-7b05"
I20240503 07:59:10.005674 1251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:10.007655 1265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.008571 1251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:59:10.008670 1251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "979ac07da07f4df9a3a3e9039a9faff8"
format_stamp: "Formatted at 2024-05-03 07:59:09 on dist-test-slave-7b05"
I20240503 07:59:10.008751 1251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:10.024142 1251 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:10.024638 1251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:10.024773 1251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:10.030499 1251 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:35123
I20240503 07:59:10.030509 1317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:35123 every 8 connection(s)
I20240503 07:59:10.031383 1251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:10.033675 1318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.038209 1318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8: Bootstrap starting.
I20240503 07:59:10.039645 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 1251
I20240503 07:59:10.039743 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:10.040247 1318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.040974 1318 log.cc:824] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:10.042649 1318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8: No bootstrap required, opened a new log
I20240503 07:59:10.045516 1318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ac07da07f4df9a3a3e9039a9faff8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35123 } }
I20240503 07:59:10.045688 1318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.045745 1318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979ac07da07f4df9a3a3e9039a9faff8, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.046175 1318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ac07da07f4df9a3a3e9039a9faff8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35123 } }
I20240503 07:59:10.046315 1318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:10.046380 1318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:10.046480 1318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.069326 1318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ac07da07f4df9a3a3e9039a9faff8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35123 } }
I20240503 07:59:10.069727 1318 leader_election.cc:302] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 979ac07da07f4df9a3a3e9039a9faff8; no voters:
I20240503 07:59:10.070029 1318 leader_election.cc:288] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:10.070133 1323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:10.070365 1323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [term 1 LEADER]: Becoming Leader. State: Replica: 979ac07da07f4df9a3a3e9039a9faff8, State: Running, Role: LEADER
I20240503 07:59:10.070884 1323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ac07da07f4df9a3a3e9039a9faff8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35123 } }
I20240503 07:59:10.071231 1318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:10.072525 1324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "979ac07da07f4df9a3a3e9039a9faff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ac07da07f4df9a3a3e9039a9faff8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35123 } } }
I20240503 07:59:10.072533 1325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 979ac07da07f4df9a3a3e9039a9faff8. Latest consensus state: current_term: 1 leader_uuid: "979ac07da07f4df9a3a3e9039a9faff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979ac07da07f4df9a3a3e9039a9faff8" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35123 } } }
I20240503 07:59:10.072644 1324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:10.072669 1325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:10.074052 1332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:10.076542 1332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:10.081053 1332 catalog_manager.cc:1347] Generated new cluster ID: eca272ac0c5a441ba61d94ba4fc9d35d
I20240503 07:59:10.081140 1332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:10.088070 1332 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:10.088825 1332 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:10.094579 1332 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 979ac07da07f4df9a3a3e9039a9faff8: Generated new TSK 0
I20240503 07:59:10.095165 1332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:10.104789 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--builtin_ntp_servers=127.0.99.148:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:10.205950 1342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:10.206172 1342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:10.206233 1342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:10.209306 1342 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:10.209367 1342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:10.209441 1342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:10.212919 1342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--never_fsync=true
--heap_profile_path=/tmp/kudu.1342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:10.213662 1342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:10.214663 1342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:10.223094 1347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.223575 1350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.224397 1348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:10.225135 1342 server_base.cc:1034] running on GCE node
I20240503 07:59:10.225451 1342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:10.225924 1342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:10.227066 1342 hybrid_clock.cc:648] HybridClock initialized: now 1714723150227054 us; error 23 us; skew 500 ppm
I20240503 07:59:10.228646 1342 webserver.cc:460] Webserver started at http://127.0.99.129:33461/ using document root <none> and password file <none>
I20240503 07:59:10.229173 1342 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:10.229225 1342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:10.229386 1342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:10.230845 1342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ebbca251d61948cdb859ef60a8b571a8"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.231273 1342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ebbca251d61948cdb859ef60a8b571a8"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.234174 1342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:10.236064 1356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.236961 1342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:10.237074 1342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ebbca251d61948cdb859ef60a8b571a8"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.237152 1342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:10.250074 1342 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:10.250538 1342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:10.250674 1342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:10.251125 1342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:10.251891 1342 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:10.251935 1342 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.251981 1342 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:10.252007 1342 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.259131 1342 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33545
I20240503 07:59:10.259162 1469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33545 every 8 connection(s)
I20240503 07:59:10.259896 1342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:10.267148 1470 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35123
I20240503 07:59:10.267328 1470 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:10.267753 1470 heartbeater.cc:511] Master 127.0.99.190:35123 requested a full tablet report, sending...
I20240503 07:59:10.268993 1282 ts_manager.cc:194] Registered new tserver with Master: ebbca251d61948cdb859ef60a8b571a8 (127.0.99.129:33545)
I20240503 07:59:10.269513 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 1342
I20240503 07:59:10.269609 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:10.270359 1282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:53591
I20240503 07:59:10.271476 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--builtin_ntp_servers=127.0.99.148:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:10.368950 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:10.369158 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:10.369204 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:10.372148 1473 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:10.372201 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:10.372279 1473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:10.376036 1473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:10.376796 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:10.377826 1473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:10.385525 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.386301 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.387045 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:10.387601 1473 server_base.cc:1034] running on GCE node
I20240503 07:59:10.387903 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:10.388363 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:10.389494 1473 hybrid_clock.cc:648] HybridClock initialized: now 1714723150389484 us; error 24 us; skew 500 ppm
I20240503 07:59:10.391147 1473 webserver.cc:460] Webserver started at http://127.0.99.130:39925/ using document root <none> and password file <none>
I20240503 07:59:10.391618 1473 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:10.391674 1473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:10.391837 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:10.393366 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "86968ceca7524d3da8b238d9fc3e720a"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.393782 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "86968ceca7524d3da8b238d9fc3e720a"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.396798 1473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:10.399183 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.400209 1473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:10.400305 1473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "86968ceca7524d3da8b238d9fc3e720a"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.400379 1473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:10.418534 1473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:10.419014 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:10.419131 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:10.419575 1473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:10.420378 1473 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:10.420421 1473 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.420455 1473 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:10.420475 1473 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.428180 1473 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46843
I20240503 07:59:10.428228 1600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46843 every 8 connection(s)
I20240503 07:59:10.429004 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:10.436086 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 1473
I20240503 07:59:10.436185 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:10.436601 1601 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35123
I20240503 07:59:10.436798 1601 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:10.437328 1601 heartbeater.cc:511] Master 127.0.99.190:35123 requested a full tablet report, sending...
I20240503 07:59:10.438031 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--builtin_ntp_servers=127.0.99.148:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:10.438190 1281 ts_manager.cc:194] Registered new tserver with Master: 86968ceca7524d3da8b238d9fc3e720a (127.0.99.130:46843)
I20240503 07:59:10.438694 1281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:59183
W20240503 07:59:10.538868 1604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:10.539052 1604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:10.539095 1604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:10.542038 1604 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:10.542090 1604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:10.542166 1604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:10.545522 1604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--never_fsync=true
--heap_profile_path=/tmp/kudu.1604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:10.546229 1604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:10.547201 1604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:10.554944 1612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.555713 1610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.556336 1609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:10.556876 1604 server_base.cc:1034] running on GCE node
I20240503 07:59:10.557217 1604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:10.557688 1604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:10.558825 1604 hybrid_clock.cc:648] HybridClock initialized: now 1714723150558807 us; error 29 us; skew 500 ppm
I20240503 07:59:10.560391 1604 webserver.cc:460] Webserver started at http://127.0.99.131:34531/ using document root <none> and password file <none>
I20240503 07:59:10.560863 1604 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:10.560914 1604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:10.561117 1604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:10.562563 1604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "440a6f2e32ad4a08bbd980b44908b1d0"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.562978 1604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "440a6f2e32ad4a08bbd980b44908b1d0"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.565873 1604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 07:59:10.567713 1618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.568617 1604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:10.568702 1604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "440a6f2e32ad4a08bbd980b44908b1d0"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.568778 1604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:10.601707 1604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:10.602274 1604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:10.602411 1604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:10.602897 1604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:10.603768 1604 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:10.603822 1604 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.603875 1604 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:10.603900 1604 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.611547 1604 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:36219
I20240503 07:59:10.611559 1731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:36219 every 8 connection(s)
I20240503 07:59:10.612406 1604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:10.612565 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 1604
I20240503 07:59:10.612653 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:10.614812 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--builtin_ntp_servers=127.0.99.148:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:10.620765 1732 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35123
I20240503 07:59:10.620955 1732 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:10.621484 1732 heartbeater.cc:511] Master 127.0.99.190:35123 requested a full tablet report, sending...
I20240503 07:59:10.622397 1281 ts_manager.cc:194] Registered new tserver with Master: 440a6f2e32ad4a08bbd980b44908b1d0 (127.0.99.131:36219)
I20240503 07:59:10.623087 1281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:53505
W20240503 07:59:10.714291 1735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:10.714460 1735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:10.714499 1735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:10.717342 1735 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:10.717392 1735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:10.717451 1735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:10.720790 1735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--never_fsync=true
--heap_profile_path=/tmp/kudu.1735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:10.721485 1735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:10.722436 1735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:10.730132 1740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.730937 1741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.731536 1743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:10.732106 1735 server_base.cc:1034] running on GCE node
I20240503 07:59:10.732422 1735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:10.732892 1735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:10.734028 1735 hybrid_clock.cc:648] HybridClock initialized: now 1714723150734016 us; error 30 us; skew 500 ppm
I20240503 07:59:10.735590 1735 webserver.cc:460] Webserver started at http://127.0.99.132:34489/ using document root <none> and password file <none>
I20240503 07:59:10.736100 1735 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:10.736155 1735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:10.736305 1735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:10.737787 1735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "09e3df902ff54236ab2bd2f5083dee8e"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.738198 1735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "09e3df902ff54236ab2bd2f5083dee8e"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.741119 1735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240503 07:59:10.743029 1749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.743881 1735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:10.743970 1735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "09e3df902ff54236ab2bd2f5083dee8e"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.744038 1735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:10.761256 1735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:10.761763 1735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:10.761898 1735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:10.762342 1735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:10.763131 1735 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:10.763176 1735 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.763226 1735 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:10.763253 1735 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.770891 1735 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43859
I20240503 07:59:10.770934 1862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43859 every 8 connection(s)
I20240503 07:59:10.771739 1735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:10.779425 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 1735
I20240503 07:59:10.779531 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:59:10.779597 1863 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35123
I20240503 07:59:10.779790 1863 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:10.780284 1863 heartbeater.cc:511] Master 127.0.99.190:35123 requested a full tablet report, sending...
I20240503 07:59:10.781189 1281 ts_manager.cc:194] Registered new tserver with Master: 09e3df902ff54236ab2bd2f5083dee8e (127.0.99.132:43859)
I20240503 07:59:10.781445 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--builtin_ntp_servers=127.0.99.148:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:10.781737 1281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:41523
W20240503 07:59:10.879185 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:10.879364 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:10.879417 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:10.882302 1866 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:10.882360 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:10.882438 1866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:10.885831 1866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35123
--never_fsync=true
--heap_profile_path=/tmp/kudu.1866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:10.886551 1866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:10.887524 1866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:10.895207 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.895768 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:10.896590 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:10.897337 1866 server_base.cc:1034] running on GCE node
I20240503 07:59:10.897655 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:10.898109 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:10.899252 1866 hybrid_clock.cc:648] HybridClock initialized: now 1714723150899241 us; error 24 us; skew 500 ppm
I20240503 07:59:10.900857 1866 webserver.cc:460] Webserver started at http://127.0.99.133:33819/ using document root <none> and password file <none>
I20240503 07:59:10.901376 1866 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:10.901432 1866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:10.901585 1866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:10.903064 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c7ba701c8bbb45ca95609b3af0caf707"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.903481 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c7ba701c8bbb45ca95609b3af0caf707"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.906486 1866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:10.908360 1880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.909278 1866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:10.909370 1866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c7ba701c8bbb45ca95609b3af0caf707"
format_stamp: "Formatted at 2024-05-03 07:59:10 on dist-test-slave-7b05"
I20240503 07:59:10.909440 1866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:10.936372 1866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:10.936913 1866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:10.937083 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:10.937567 1866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:10.938373 1866 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:10.938417 1866 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.938467 1866 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:10.938491 1866 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:10.945892 1866 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:40125
I20240503 07:59:10.945917 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:40125 every 8 connection(s)
I20240503 07:59:10.946729 1866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:10.954246 1994 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35123
I20240503 07:59:10.954432 1994 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:10.954866 1994 heartbeater.cc:511] Master 127.0.99.190:35123 requested a full tablet report, sending...
I20240503 07:59:10.955657 1281 ts_manager.cc:194] Registered new tserver with Master: c7ba701c8bbb45ca95609b3af0caf707 (127.0.99.133:40125)
I20240503 07:59:10.956144 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 1866
I20240503 07:59:10.956154 1281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:48049
I20240503 07:59:10.956225 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:59:10.958113 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:59:10.965430 1281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35612:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:59:10.981336 1535 tablet_service.cc:1467] Processing CreateTablet for tablet c0f96201b3964ede8c0d1b45c161e2c7 (DEFAULT_TABLE table=TestTable [id=3dd6c58788c54fbb967b0e3b69814258]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.981318 1666 tablet_service.cc:1467] Processing CreateTablet for tablet c0f96201b3964ede8c0d1b45c161e2c7 (DEFAULT_TABLE table=TestTable [id=3dd6c58788c54fbb967b0e3b69814258]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.981335 1404 tablet_service.cc:1467] Processing CreateTablet for tablet c0f96201b3964ede8c0d1b45c161e2c7 (DEFAULT_TABLE table=TestTable [id=3dd6c58788c54fbb967b0e3b69814258]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.982156 1666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f96201b3964ede8c0d1b45c161e2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.982180 1535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f96201b3964ede8c0d1b45c161e2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.982180 1404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0f96201b3964ede8c0d1b45c161e2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.986675 2011 tablet_bootstrap.cc:492] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0: Bootstrap starting.
I20240503 07:59:10.987928 2012 tablet_bootstrap.cc:492] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a: Bootstrap starting.
I20240503 07:59:10.987994 2013 tablet_bootstrap.cc:492] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8: Bootstrap starting.
I20240503 07:59:10.988709 2011 tablet_bootstrap.cc:654] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.989463 2012 tablet_bootstrap.cc:654] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.989784 2013 tablet_bootstrap.cc:654] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.989979 2011 log.cc:824] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:10.990149 2012 log.cc:824] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:10.990419 2013 log.cc:824] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:10.991590 2012 tablet_bootstrap.cc:492] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a: No bootstrap required, opened a new log
I20240503 07:59:10.991590 2011 tablet_bootstrap.cc:492] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0: No bootstrap required, opened a new log
I20240503 07:59:10.991678 2013 tablet_bootstrap.cc:492] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8: No bootstrap required, opened a new log
I20240503 07:59:10.991752 2012 ts_tablet_manager.cc:1367] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:10.991752 2011 ts_tablet_manager.cc:1367] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 07:59:10.991878 2013 ts_tablet_manager.cc:1367] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:10.994397 2013 raft_consensus.cc:348] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } }
I20240503 07:59:10.994561 2013 raft_consensus.cc:374] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.994611 2013 raft_consensus.cc:729] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbca251d61948cdb859ef60a8b571a8, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.994994 2013 consensus_queue.cc:260] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } }
I20240503 07:59:10.995100 2012 raft_consensus.cc:348] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } }
I20240503 07:59:10.995283 2012 raft_consensus.cc:374] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.995333 2012 raft_consensus.cc:729] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86968ceca7524d3da8b238d9fc3e720a, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.995771 2013 ts_tablet_manager.cc:1398] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:10.995721 2012 consensus_queue.cc:260] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } }
I20240503 07:59:10.995682 2011 raft_consensus.cc:348] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } }
I20240503 07:59:10.995847 2011 raft_consensus.cc:374] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.995908 2011 raft_consensus.cc:729] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440a6f2e32ad4a08bbd980b44908b1d0, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.996088 1470 heartbeater.cc:503] Master 127.0.99.190:35123 was elected leader, sending a full tablet report...
I20240503 07:59:10.996299 2011 consensus_queue.cc:260] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } }
I20240503 07:59:10.996366 1601 heartbeater.cc:503] Master 127.0.99.190:35123 was elected leader, sending a full tablet report...
I20240503 07:59:10.996459 2012 ts_tablet_manager.cc:1398] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:10.996918 2011 ts_tablet_manager.cc:1398] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20240503 07:59:10.997207 1732 heartbeater.cc:503] Master 127.0.99.190:35123 was elected leader, sending a full tablet report...
W20240503 07:59:11.010953 1471 tablet.cc:2369] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:11.040210 2019 raft_consensus.cc:482] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.040385 2019 raft_consensus.cc:504] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } }
I20240503 07:59:11.041730 2019 leader_election.cc:288] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebbca251d61948cdb859ef60a8b571a8 (127.0.99.129:33545), 86968ceca7524d3da8b238d9fc3e720a (127.0.99.130:46843)
I20240503 07:59:11.045116 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f96201b3964ede8c0d1b45c161e2c7" candidate_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbca251d61948cdb859ef60a8b571a8" is_pre_election: true
I20240503 07:59:11.045114 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f96201b3964ede8c0d1b45c161e2c7" candidate_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86968ceca7524d3da8b238d9fc3e720a" is_pre_election: true
I20240503 07:59:11.045352 1555 raft_consensus.cc:2454] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 440a6f2e32ad4a08bbd980b44908b1d0 in term 0.
I20240503 07:59:11.045357 1424 raft_consensus.cc:2454] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 440a6f2e32ad4a08bbd980b44908b1d0 in term 0.
I20240503 07:59:11.045742 1619 leader_election.cc:302] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440a6f2e32ad4a08bbd980b44908b1d0, ebbca251d61948cdb859ef60a8b571a8; no voters:
I20240503 07:59:11.045967 2019 raft_consensus.cc:2781] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.046034 2019 raft_consensus.cc:482] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.046063 2019 raft_consensus.cc:3037] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.089310 2019 raft_consensus.cc:504] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } }
I20240503 07:59:11.089816 2019 leader_election.cc:288] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [CANDIDATE]: Term 1 election: Requested vote from peers ebbca251d61948cdb859ef60a8b571a8 (127.0.99.129:33545), 86968ceca7524d3da8b238d9fc3e720a (127.0.99.130:46843)
I20240503 07:59:11.090267 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f96201b3964ede8c0d1b45c161e2c7" candidate_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbca251d61948cdb859ef60a8b571a8"
I20240503 07:59:11.090356 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0f96201b3964ede8c0d1b45c161e2c7" candidate_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86968ceca7524d3da8b238d9fc3e720a"
I20240503 07:59:11.090404 1424 raft_consensus.cc:3037] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.090498 1555 raft_consensus.cc:3037] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.091354 1555 raft_consensus.cc:2454] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 440a6f2e32ad4a08bbd980b44908b1d0 in term 1.
I20240503 07:59:11.091468 1424 raft_consensus.cc:2454] T c0f96201b3964ede8c0d1b45c161e2c7 P ebbca251d61948cdb859ef60a8b571a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 440a6f2e32ad4a08bbd980b44908b1d0 in term 1.
I20240503 07:59:11.091746 1620 leader_election.cc:302] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440a6f2e32ad4a08bbd980b44908b1d0, 86968ceca7524d3da8b238d9fc3e720a; no voters:
I20240503 07:59:11.092070 2019 raft_consensus.cc:2781] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.092288 2019 raft_consensus.cc:686] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [term 1 LEADER]: Becoming Leader. State: Replica: 440a6f2e32ad4a08bbd980b44908b1d0, State: Running, Role: LEADER
I20240503 07:59:11.092623 2019 consensus_queue.cc:237] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } }
I20240503 07:59:11.095697 1280 catalog_manager.cc:5484] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 440a6f2e32ad4a08bbd980b44908b1d0 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "440a6f2e32ad4a08bbd980b44908b1d0" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:11.113346 1733 tablet.cc:2369] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:11.125643 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:59:11.127171 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ebbca251d61948cdb859ef60a8b571a8 to finish bootstrapping
I20240503 07:59:11.132411 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 86968ceca7524d3da8b238d9fc3e720a to finish bootstrapping
I20240503 07:59:11.136387 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 440a6f2e32ad4a08bbd980b44908b1d0 to finish bootstrapping
I20240503 07:59:11.140396 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 09e3df902ff54236ab2bd2f5083dee8e to finish bootstrapping
I20240503 07:59:11.145584 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c7ba701c8bbb45ca95609b3af0caf707 to finish bootstrapping
W20240503 07:59:11.180181 1602 tablet.cc:2369] T c0f96201b3964ede8c0d1b45c161e2c7 P 86968ceca7524d3da8b238d9fc3e720a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:11.319751 1535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.320801 1928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.321424 1797 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.321609 1666 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.322652 1404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.403867 1666 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.403889 1535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.404620 1797 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.405973 1928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.407904 1404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.503990 1666 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.504453 1535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.504555 1797 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.505551 1928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.513721 1404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.573271 2019 consensus_queue.cc:1035] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86968ceca7524d3da8b238d9fc3e720a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.578874 2021 consensus_queue.cc:1035] T c0f96201b3964ede8c0d1b45c161e2c7 P 440a6f2e32ad4a08bbd980b44908b1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbca251d61948cdb859ef60a8b571a8" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.714537 1797 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.715715 1666 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.717167 1928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.718077 1404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.735169 1535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.783519 1863 heartbeater.cc:503] Master 127.0.99.190:35123 was elected leader, sending a full tablet report...
I20240503 07:59:11.801621 1797 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.805387 1404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.806402 1666 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.807006 1535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.824518 1928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.907986 1535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.908727 1797 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.909178 1666 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.909420 1404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.909780 1928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.944902 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 1342
I20240503 07:59:11.951193 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 1473
I20240503 07:59:11.955781 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 1604
I20240503 07:59:11.958004 1994 heartbeater.cc:503] Master 127.0.99.190:35123 was elected leader, sending a full tablet report...
I20240503 07:59:11.960476 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 1735
I20240503 07:59:11.965255 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 1866
I20240503 07:59:11.969261 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 1251
2024-05-03T07:59:11Z chronyd exiting
[ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (2121 ms)
[----------] 1 test from RebalanceStartSafetyTest (2121 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/7
I20240503 07:59:11.989082 398 test_util.cc:268] Using random seed: 408734038
I20240503 07:59:11.989751 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:11.989785 398 ts_itest-base.cc:113] --------------
I20240503 07:59:11.989800 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 07:59:11.989821 398 ts_itest-base.cc:115] 5 replicas per TS
I20240503 07:59:11.989841 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:11Z Disabled control of system clock
I20240503 07:59:11.997504 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42701
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42701
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 07:59:12.095350 2236 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 07:59:12.095558 2236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:12.095594 2236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:12.095652 2236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:12.098534 2236 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:12.098600 2236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:12.098631 2236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:12.098663 2236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:12.098688 2236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:12.102154 2236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42701
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:12.102923 2236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:12.103920 2236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:12.110769 2241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.111321 2244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.112216 2242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:12.112915 2236 server_base.cc:1034] running on GCE node
I20240503 07:59:12.113404 2236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:12.114189 2236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:12.115327 2236 hybrid_clock.cc:648] HybridClock initialized: now 1714723152115317 us; error 24 us; skew 500 ppm
I20240503 07:59:12.117013 2236 webserver.cc:460] Webserver started at http://127.0.99.190:43369/ using document root <none> and password file <none>
I20240503 07:59:12.117509 2236 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:12.117563 2236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:12.117712 2236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:12.119194 2236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1a6fc308ffbf49c2b3e717d2a724d973"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.119611 2236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1a6fc308ffbf49c2b3e717d2a724d973"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.122675 2236 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:12.124500 2250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.125526 2236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:12.125622 2236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1a6fc308ffbf49c2b3e717d2a724d973"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.125694 2236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:12.134227 2236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:12.134694 2236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:12.134814 2236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:12.140784 2236 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42701
I20240503 07:59:12.140805 2302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42701 every 8 connection(s)
I20240503 07:59:12.141597 2236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:12.143774 2303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.147951 2303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973: Bootstrap starting.
I20240503 07:59:12.149847 2303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.150422 2303 log.cc:824] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:12.150971 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 2236
I20240503 07:59:12.151070 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:12.151922 2303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973: No bootstrap required, opened a new log
I20240503 07:59:12.154848 2303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6fc308ffbf49c2b3e717d2a724d973" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42701 } }
I20240503 07:59:12.155001 2303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.155061 2303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6fc308ffbf49c2b3e717d2a724d973, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.155412 2303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6fc308ffbf49c2b3e717d2a724d973" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42701 } }
I20240503 07:59:12.155520 2303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:12.155571 2303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:12.155658 2303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.157738 2303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6fc308ffbf49c2b3e717d2a724d973" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42701 } }
I20240503 07:59:12.158052 2303 leader_election.cc:302] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a6fc308ffbf49c2b3e717d2a724d973; no voters:
I20240503 07:59:12.158305 2303 leader_election.cc:288] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:12.158854 2309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.159025 2309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [term 1 LEADER]: Becoming Leader. State: Replica: 1a6fc308ffbf49c2b3e717d2a724d973, State: Running, Role: LEADER
I20240503 07:59:12.159127 2303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:12.159273 2309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6fc308ffbf49c2b3e717d2a724d973" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42701 } }
I20240503 07:59:12.160277 2309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a6fc308ffbf49c2b3e717d2a724d973. Latest consensus state: current_term: 1 leader_uuid: "1a6fc308ffbf49c2b3e717d2a724d973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6fc308ffbf49c2b3e717d2a724d973" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42701 } } }
I20240503 07:59:12.160393 2309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:12.160569 2310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a6fc308ffbf49c2b3e717d2a724d973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a6fc308ffbf49c2b3e717d2a724d973" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42701 } } }
I20240503 07:59:12.160653 2310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:12.160688 2317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:12.163393 2317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:12.167584 2317 catalog_manager.cc:1347] Generated new cluster ID: 921f44e20a294b379d9b89bc9ec5fc94
I20240503 07:59:12.167641 2317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:12.186982 2317 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:12.188050 2317 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:12.200026 2317 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1a6fc308ffbf49c2b3e717d2a724d973: Generated new TSK 0
I20240503 07:59:12.200556 2317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:12.208712 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:12.305964 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:12.306135 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:12.306180 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:12.309013 2327 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:12.309104 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:12.309191 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:12.312544 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:12.313277 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:12.314221 2327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:12.321513 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.322021 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.322924 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:12.323666 2327 server_base.cc:1034] running on GCE node
I20240503 07:59:12.323968 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:12.324409 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:12.325543 2327 hybrid_clock.cc:648] HybridClock initialized: now 1714723152325523 us; error 33 us; skew 500 ppm
I20240503 07:59:12.327128 2327 webserver.cc:460] Webserver started at http://127.0.99.129:43271/ using document root <none> and password file <none>
I20240503 07:59:12.327594 2327 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:12.327647 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:12.327795 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:12.329304 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.329715 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.333194 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:12.335374 2341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.336231 2327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:12.336323 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.336398 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:12.351361 2327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:12.351873 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:12.351998 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:12.352445 2327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:12.353246 2327 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:12.353291 2327 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.353327 2327 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:12.353355 2327 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.361027 2327 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:38251
I20240503 07:59:12.361089 2454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:38251 every 8 connection(s)
I20240503 07:59:12.361835 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:12.362993 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 2327
I20240503 07:59:12.363095 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:12.364954 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:12.369971 2455 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42701
I20240503 07:59:12.370157 2455 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:12.370591 2455 heartbeater.cc:511] Master 127.0.99.190:42701 requested a full tablet report, sending...
I20240503 07:59:12.371977 2267 ts_manager.cc:194] Registered new tserver with Master: cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251)
I20240503 07:59:12.373251 2267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:48671
W20240503 07:59:12.466410 2458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:12.466905 2458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:12.467031 2458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:12.472509 2458 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:12.472571 2458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:12.472649 2458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:12.476500 2458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--never_fsync=true
--heap_profile_path=/tmp/kudu.2458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:12.477324 2458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:12.478330 2458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:12.486239 2463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.486745 2464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.487632 2466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:12.488360 2458 server_base.cc:1034] running on GCE node
I20240503 07:59:12.488718 2458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:12.489264 2458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:12.490399 2458 hybrid_clock.cc:648] HybridClock initialized: now 1714723152490387 us; error 23 us; skew 500 ppm
I20240503 07:59:12.492092 2458 webserver.cc:460] Webserver started at http://127.0.99.130:40691/ using document root <none> and password file <none>
I20240503 07:59:12.492585 2458 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:12.492635 2458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:12.492795 2458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:12.494342 2458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3631cd1b9b79487581e9d31178f671c3"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.494764 2458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3631cd1b9b79487581e9d31178f671c3"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.497668 2458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:12.499550 2472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.500666 2458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:12.500779 2458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3631cd1b9b79487581e9d31178f671c3"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.500864 2458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:12.511622 2458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:12.512128 2458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:12.512259 2458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:12.512710 2458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:12.513520 2458 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:12.513561 2458 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.513599 2458 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:12.513626 2458 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.521335 2458 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:40355
I20240503 07:59:12.521366 2585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:40355 every 8 connection(s)
I20240503 07:59:12.522114 2458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:12.529841 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 2458
I20240503 07:59:12.529881 2586 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42701
I20240503 07:59:12.529958 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:12.530079 2586 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:12.530521 2586 heartbeater.cc:511] Master 127.0.99.190:42701 requested a full tablet report, sending...
I20240503 07:59:12.531431 2267 ts_manager.cc:194] Registered new tserver with Master: 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355)
I20240503 07:59:12.532083 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:12.532122 2267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:40523
W20240503 07:59:12.634168 2589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:12.634357 2589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:12.634413 2589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:12.637349 2589 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:12.637404 2589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:12.637480 2589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:12.641077 2589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--never_fsync=true
--heap_profile_path=/tmp/kudu.2589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:12.641867 2589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:12.642863 2589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:12.651649 2594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.652456 2595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.653191 2597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:12.653805 2589 server_base.cc:1034] running on GCE node
I20240503 07:59:12.654084 2589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:12.654536 2589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:12.655679 2589 hybrid_clock.cc:648] HybridClock initialized: now 1714723152655658 us; error 30 us; skew 500 ppm
I20240503 07:59:12.657389 2589 webserver.cc:460] Webserver started at http://127.0.99.131:37893/ using document root <none> and password file <none>
I20240503 07:59:12.657878 2589 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:12.657934 2589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:12.658083 2589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:12.659577 2589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "886e4eb306064b98ab923623b8f1d401"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.660013 2589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "886e4eb306064b98ab923623b8f1d401"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.663050 2589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:12.664893 2603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.665799 2589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:12.665899 2589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "886e4eb306064b98ab923623b8f1d401"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.665969 2589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:12.686133 2589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:12.686654 2589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:12.686780 2589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:12.687222 2589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:12.687986 2589 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:12.688030 2589 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.688076 2589 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:12.688100 2589 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.695489 2589 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44343
I20240503 07:59:12.695516 2716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44343 every 8 connection(s)
I20240503 07:59:12.696267 2589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:12.696760 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 2589
I20240503 07:59:12.696853 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:12.698948 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:12.704597 2717 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42701
I20240503 07:59:12.704835 2717 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:12.705468 2717 heartbeater.cc:511] Master 127.0.99.190:42701 requested a full tablet report, sending...
I20240503 07:59:12.706473 2267 ts_manager.cc:194] Registered new tserver with Master: 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:12.707110 2267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:58653
W20240503 07:59:12.800158 2720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:12.800346 2720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:12.800402 2720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:12.803444 2720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:12.803500 2720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:12.803579 2720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:12.807098 2720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--never_fsync=true
--heap_profile_path=/tmp/kudu.2720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:12.807835 2720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:12.808831 2720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:12.816593 2725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.817138 2726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.817973 2728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:12.818725 2720 server_base.cc:1034] running on GCE node
I20240503 07:59:12.819037 2720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:12.819512 2720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:12.820676 2720 hybrid_clock.cc:648] HybridClock initialized: now 1714723152820654 us; error 31 us; skew 500 ppm
I20240503 07:59:12.822398 2720 webserver.cc:460] Webserver started at http://127.0.99.132:33349/ using document root <none> and password file <none>
I20240503 07:59:12.822866 2720 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:12.822921 2720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:12.823064 2720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:12.824530 2720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0bc088d2657a49b39cb7981f0e3eec21"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.824944 2720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0bc088d2657a49b39cb7981f0e3eec21"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.827829 2720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:12.829838 2734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.830711 2720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:12.830796 2720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0bc088d2657a49b39cb7981f0e3eec21"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.830859 2720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:12.862214 2720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:12.862757 2720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:12.862879 2720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:12.863322 2720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:12.864089 2720 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:12.864131 2720 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.864166 2720 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:12.864184 2720 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:12.871793 2720 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37199
I20240503 07:59:12.871824 2847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37199 every 8 connection(s)
I20240503 07:59:12.872601 2720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:12.873953 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 2720
I20240503 07:59:12.874075 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:59:12.875952 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:12.880826 2848 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42701
I20240503 07:59:12.881017 2848 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:12.881493 2848 heartbeater.cc:511] Master 127.0.99.190:42701 requested a full tablet report, sending...
I20240503 07:59:12.882325 2267 ts_manager.cc:194] Registered new tserver with Master: 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199)
I20240503 07:59:12.882826 2267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:55643
W20240503 07:59:12.974262 2851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:12.974427 2851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:12.974467 2851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:12.977371 2851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:12.977424 2851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:12.977499 2851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:12.980890 2851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--never_fsync=true
--heap_profile_path=/tmp/kudu.2851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:12.981663 2851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:12.982650 2851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:12.990413 2856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.990876 2859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:12.991389 2857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:12.992345 2851 server_base.cc:1034] running on GCE node
I20240503 07:59:12.992659 2851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:12.993158 2851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:12.994294 2851 hybrid_clock.cc:648] HybridClock initialized: now 1714723152994285 us; error 26 us; skew 500 ppm
I20240503 07:59:12.996047 2851 webserver.cc:460] Webserver started at http://127.0.99.133:45127/ using document root <none> and password file <none>
I20240503 07:59:12.996532 2851 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:12.996587 2851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:12.996747 2851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:12.998293 2851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e3b61927599645d4b11278833b2f27d6"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:12.998715 2851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e3b61927599645d4b11278833b2f27d6"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:13.001674 2851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 07:59:13.003541 2865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:13.004426 2851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:13.004515 2851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e3b61927599645d4b11278833b2f27d6"
format_stamp: "Formatted at 2024-05-03 07:59:12 on dist-test-slave-7b05"
I20240503 07:59:13.004583 2851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:13.025331 2851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:13.025843 2851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:13.025976 2851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:13.026415 2851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:13.027194 2851 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:13.027238 2851 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:13.027284 2851 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:13.027308 2851 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:13.035130 2851 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:46695
I20240503 07:59:13.035152 2978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:46695 every 8 connection(s)
I20240503 07:59:13.035895 2851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:13.041147 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 2851
I20240503 07:59:13.041242 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:59:13.043112 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:13.043615 2979 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42701
I20240503 07:59:13.043814 2979 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:13.044358 2979 heartbeater.cc:511] Master 127.0.99.190:42701 requested a full tablet report, sending...
I20240503 07:59:13.045215 2267 ts_manager.cc:194] Registered new tserver with Master: e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:13.045725 2267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:54957
W20240503 07:59:13.140775 2982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:13.140942 2982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:13.140995 2982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:13.143844 2982 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:13.143903 2982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:13.143982 2982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:13.147338 2982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--never_fsync=true
--heap_profile_path=/tmp/kudu.2982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:13.148012 2982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:13.148945 2982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:13.156543 2990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:13.157305 2987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:13.157899 2988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:13.158509 2982 server_base.cc:1034] running on GCE node
I20240503 07:59:13.158834 2982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:13.159329 2982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:13.160472 2982 hybrid_clock.cc:648] HybridClock initialized: now 1714723153160464 us; error 25 us; skew 500 ppm
I20240503 07:59:13.162642 2982 webserver.cc:460] Webserver started at http://127.0.99.134:33119/ using document root <none> and password file <none>
I20240503 07:59:13.163136 2982 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:13.163190 2982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:13.163339 2982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:13.164863 2982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "81b9495629bb458892b031db80bc3715"
format_stamp: "Formatted at 2024-05-03 07:59:13 on dist-test-slave-7b05"
I20240503 07:59:13.165297 2982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "81b9495629bb458892b031db80bc3715"
format_stamp: "Formatted at 2024-05-03 07:59:13 on dist-test-slave-7b05"
I20240503 07:59:13.168197 2982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:13.170221 2996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:13.171099 2982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:13.171187 2982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "81b9495629bb458892b031db80bc3715"
format_stamp: "Formatted at 2024-05-03 07:59:13 on dist-test-slave-7b05"
I20240503 07:59:13.171270 2982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:13.191188 2982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:13.191689 2982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:13.191817 2982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:13.192271 2982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:13.193038 2982 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:13.193127 2982 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:13.193177 2982 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:13.193202 2982 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:13.201793 2982 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:36145
I20240503 07:59:13.201825 3109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:36145 every 8 connection(s)
I20240503 07:59:13.202579 2982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:13.207602 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 2982
I20240503 07:59:13.207720 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:59:13.210099 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:13.210405 3110 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42701
I20240503 07:59:13.210588 3110 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:13.211028 3110 heartbeater.cc:511] Master 127.0.99.190:42701 requested a full tablet report, sending...
I20240503 07:59:13.211879 2267 ts_manager.cc:194] Registered new tserver with Master: 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:13.212385 2267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:49731
W20240503 07:59:13.309029 3113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:13.309221 3113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:13.309264 3113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:13.312232 3113 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:13.312289 3113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:13.312407 3113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:13.315824 3113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--never_fsync=true
--heap_profile_path=/tmp/kudu.3113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:13.316542 3113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:13.317534 3113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:13.325268 3118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:13.325765 3119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:13.326612 3121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:13.327327 3113 server_base.cc:1034] running on GCE node
I20240503 07:59:13.327597 3113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:13.328028 3113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:13.329159 3113 hybrid_clock.cc:648] HybridClock initialized: now 1714723153329149 us; error 28 us; skew 500 ppm
I20240503 07:59:13.330787 3113 webserver.cc:460] Webserver started at http://127.0.99.135:40585/ using document root <none> and password file <none>
I20240503 07:59:13.331290 3113 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:13.331357 3113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:13.331537 3113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:13.332967 3113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6338e74c39eb4583842e387e64fce522"
format_stamp: "Formatted at 2024-05-03 07:59:13 on dist-test-slave-7b05"
I20240503 07:59:13.333431 3113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6338e74c39eb4583842e387e64fce522"
format_stamp: "Formatted at 2024-05-03 07:59:13 on dist-test-slave-7b05"
I20240503 07:59:13.336242 3113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:13.338075 3127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:13.338912 3113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:13.339002 3113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6338e74c39eb4583842e387e64fce522"
format_stamp: "Formatted at 2024-05-03 07:59:13 on dist-test-slave-7b05"
I20240503 07:59:13.339077 3113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:13.357259 3113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:13.357772 3113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:13.357913 3113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:13.358474 3113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:13.359282 3113 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:13.359333 3113 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:13.359377 3113 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:13.359398 3113 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:13.368343 3113 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:38745
I20240503 07:59:13.368377 3240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:38745 every 8 connection(s)
I20240503 07:59:13.369204 3113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:13.374634 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 3113
I20240503 07:59:13.374743 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:59:13.375146 2455 heartbeater.cc:503] Master 127.0.99.190:42701 was elected leader, sending a full tablet report...
I20240503 07:59:13.377465 3241 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42701
I20240503 07:59:13.377635 3241 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:13.378060 3241 heartbeater.cc:511] Master 127.0.99.190:42701 requested a full tablet report, sending...
I20240503 07:59:13.378810 2266 ts_manager.cc:194] Registered new tserver with Master: 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745)
I20240503 07:59:13.379299 2266 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:46997
I20240503 07:59:13.388639 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:59:13.396191 2266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45856:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:59:13.412307 2913 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad0b76f04404591b8e4947bbc6f9d73 (DEFAULT_TABLE table=TestTable [id=9fda7ceb7e054b4c906909f1f94103be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.413209 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad0b76f04404591b8e4947bbc6f9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.413025 2782 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad0b76f04404591b8e4947bbc6f9d73 (DEFAULT_TABLE table=TestTable [id=9fda7ceb7e054b4c906909f1f94103be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.413830 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad0b76f04404591b8e4947bbc6f9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.413920 3044 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad0b76f04404591b8e4947bbc6f9d73 (DEFAULT_TABLE table=TestTable [id=9fda7ceb7e054b4c906909f1f94103be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.414039 2651 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad0b76f04404591b8e4947bbc6f9d73 (DEFAULT_TABLE table=TestTable [id=9fda7ceb7e054b4c906909f1f94103be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.414664 3044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad0b76f04404591b8e4947bbc6f9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.414798 2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad0b76f04404591b8e4947bbc6f9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.419678 3263 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:13.420166 3264 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:13.420629 3266 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:13.421922 3263 tablet_bootstrap.cc:654] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.422044 3264 tablet_bootstrap.cc:654] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.422291 3266 tablet_bootstrap.cc:654] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.422547 3263 log.cc:824] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:13.422704 3264 log.cc:824] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:13.422869 3266 log.cc:824] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:13.424001 3264 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:13.424001 3263 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:13.424160 3263 ts_tablet_manager.cc:1367] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:13.424268 3266 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:13.424410 3266 ts_tablet_manager.cc:1367] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:13.424176 3264 ts_tablet_manager.cc:1367] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:13.427199 3266 raft_consensus.cc:348] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.427423 3266 raft_consensus.cc:374] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.427484 3266 raft_consensus.cc:729] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.427541 3263 raft_consensus.cc:348] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.427723 3263 raft_consensus.cc:374] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.427771 3263 raft_consensus.cc:729] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.427909 3266 consensus_queue.cc:260] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.428165 3263 consensus_queue.cc:260] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.428159 3264 raft_consensus.cc:348] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.428453 3264 raft_consensus.cc:374] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.428534 3264 raft_consensus.cc:729] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.428716 3266 ts_tablet_manager.cc:1398] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 07:59:13.428900 3263 ts_tablet_manager.cc:1398] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:13.429026 2717 heartbeater.cc:503] Master 127.0.99.190:42701 was elected leader, sending a full tablet report...
I20240503 07:59:13.429033 3264 consensus_queue.cc:260] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.429214 2979 heartbeater.cc:503] Master 127.0.99.190:42701 was elected leader, sending a full tablet report...
I20240503 07:59:13.429574 3175 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad0b76f04404591b8e4947bbc6f9d73 (DEFAULT_TABLE table=TestTable [id=9fda7ceb7e054b4c906909f1f94103be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.430351 3175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad0b76f04404591b8e4947bbc6f9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.430655 3264 ts_tablet_manager.cc:1398] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240503 07:59:13.430958 2848 heartbeater.cc:503] Master 127.0.99.190:42701 was elected leader, sending a full tablet report...
I20240503 07:59:13.432191 3265 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:13.434000 3265 tablet_bootstrap.cc:654] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.435129 3265 log.cc:824] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:13.435272 3273 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:13.436539 3265 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:13.436693 3265 ts_tablet_manager.cc:1367] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:13.437192 3273 tablet_bootstrap.cc:654] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.437732 3273 log.cc:824] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:13.438874 3273 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: No bootstrap required, opened a new log
I20240503 07:59:13.439015 3273 ts_tablet_manager.cc:1367] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:13.439350 3265 raft_consensus.cc:348] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.439515 3265 raft_consensus.cc:374] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.439566 3265 raft_consensus.cc:729] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.439904 3265 consensus_queue.cc:260] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.440505 3110 heartbeater.cc:503] Master 127.0.99.190:42701 was elected leader, sending a full tablet report...
I20240503 07:59:13.440603 3265 ts_tablet_manager.cc:1398] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:13.441917 3273 raft_consensus.cc:348] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.442104 3273 raft_consensus.cc:374] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.442159 3273 raft_consensus.cc:729] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.442528 3273 consensus_queue.cc:260] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.443111 3241 heartbeater.cc:503] Master 127.0.99.190:42701 was elected leader, sending a full tablet report...
I20240503 07:59:13.443183 3273 ts_tablet_manager.cc:1398] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
W20240503 07:59:13.447520 2718 tablet.cc:2369] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:13.453267 3111 tablet.cc:2369] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:13.534047 2586 heartbeater.cc:503] Master 127.0.99.190:42701 was elected leader, sending a full tablet report...
W20240503 07:59:13.536788 2980 tablet.cc:2369] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:13.607954 3276 raft_consensus.cc:482] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.608170 3276 raft_consensus.cc:504] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.609361 3276 leader_election.cc:288] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745)
I20240503 07:59:13.614418 3195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad0b76f04404591b8e4947bbc6f9d73" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6338e74c39eb4583842e387e64fce522" is_pre_election: true
I20240503 07:59:13.614435 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad0b76f04404591b8e4947bbc6f9d73" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:13.614696 2933 raft_consensus.cc:2454] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:13.614697 3195 raft_consensus.cc:2454] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:13.614809 2671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad0b76f04404591b8e4947bbc6f9d73" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:13.615024 2671 raft_consensus.cc:2454] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:13.615053 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad0b76f04404591b8e4947bbc6f9d73" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:13.615160 2998 leader_election.cc:302] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 6338e74c39eb4583842e387e64fce522, 81b9495629bb458892b031db80bc3715, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:13.615232 2802 raft_consensus.cc:2454] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:13.615396 3276 raft_consensus.cc:2781] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.615470 3276 raft_consensus.cc:482] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.615517 3276 raft_consensus.cc:3037] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.616308 3276 raft_consensus.cc:504] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:13.616694 3276 leader_election.cc:288] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 election: Requested vote from peers e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745)
I20240503 07:59:13.616914 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad0b76f04404591b8e4947bbc6f9d73" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:13.616935 3195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad0b76f04404591b8e4947bbc6f9d73" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6338e74c39eb4583842e387e64fce522"
I20240503 07:59:13.616945 2671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad0b76f04404591b8e4947bbc6f9d73" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:13.616967 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad0b76f04404591b8e4947bbc6f9d73" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:13.617029 3195 raft_consensus.cc:3037] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.617026 2802 raft_consensus.cc:3037] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.617048 2933 raft_consensus.cc:3037] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.617076 2671 raft_consensus.cc:3037] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.617996 3195 raft_consensus.cc:2454] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:13.617996 2671 raft_consensus.cc:2454] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:13.618034 2933 raft_consensus.cc:2454] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:13.618036 2802 raft_consensus.cc:2454] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:13.618300 2999 leader_election.cc:302] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 6338e74c39eb4583842e387e64fce522, 81b9495629bb458892b031db80bc3715, 886e4eb306064b98ab923623b8f1d401; no voters:
I20240503 07:59:13.618428 3276 raft_consensus.cc:2781] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.618593 3276 raft_consensus.cc:686] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Becoming Leader. State: Replica: 81b9495629bb458892b031db80bc3715, State: Running, Role: LEADER
I20240503 07:59:13.619050 3276 consensus_queue.cc:237] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
W20240503 07:59:13.619931 3242 tablet.cc:2369] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:13.621903 2267 catalog_manager.cc:5484] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81b9495629bb458892b031db80bc3715 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:13.623644 2849 tablet.cc:2369] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:13.638857 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:59:13.640085 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cd82dbc60d0f40688fbe8e712dc7107a to finish bootstrapping
I20240503 07:59:13.647722 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3631cd1b9b79487581e9d31178f671c3 to finish bootstrapping
I20240503 07:59:13.654223 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 886e4eb306064b98ab923623b8f1d401 to finish bootstrapping
I20240503 07:59:13.659360 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0bc088d2657a49b39cb7981f0e3eec21 to finish bootstrapping
I20240503 07:59:13.664198 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e3b61927599645d4b11278833b2f27d6 to finish bootstrapping
I20240503 07:59:13.668061 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 81b9495629bb458892b031db80bc3715 to finish bootstrapping
I20240503 07:59:13.672055 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6338e74c39eb4583842e387e64fce522 to finish bootstrapping
I20240503 07:59:13.676041 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 3113
W20240503 07:59:14.050568 2998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38745: connect: Connection refused (error 111)
W20240503 07:59:14.051394 2998 consensus_peers.cc:485] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38745: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:14.094362 3276 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.108057 3276 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.112797 3281 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:16.552436 2998 consensus_peers.cc:485] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38745: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:17.431959 2267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45856:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:17.438783 2782 tablet_service.cc:1467] Processing CreateTablet for tablet ccac60c83bd54650a50a07d87e3fc72a (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.439147 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccac60c83bd54650a50a07d87e3fc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.439807 2650 tablet_service.cc:1467] Processing CreateTablet for tablet 66e126a09c874c51a4c83ccc54ebc2ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.440119 2650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e126a09c874c51a4c83ccc54ebc2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.441545 2781 tablet_service.cc:1467] Processing CreateTablet for tablet 66e126a09c874c51a4c83ccc54ebc2ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.441792 2781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e126a09c874c51a4c83ccc54ebc2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.441927 2913 tablet_service.cc:1467] Processing CreateTablet for tablet ccac60c83bd54650a50a07d87e3fc72a (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.442195 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccac60c83bd54650a50a07d87e3fc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.439810 2651 tablet_service.cc:1467] Processing CreateTablet for tablet ccac60c83bd54650a50a07d87e3fc72a (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.442633 2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccac60c83bd54650a50a07d87e3fc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.442709 3312 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:17.443693 3312 tablet_bootstrap.cc:654] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.443948 2650 tablet_service.cc:1467] Processing CreateTablet for tablet 71eeb5740a7344749b534273ba1c9922 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.444175 2650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71eeb5740a7344749b534273ba1c9922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.444777 3311 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:17.444831 3312 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:17.444901 3312 ts_tablet_manager.cc:1367] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:17.445030 3044 tablet_service.cc:1467] Processing CreateTablet for tablet 66e126a09c874c51a4c83ccc54ebc2ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.445308 3044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e126a09c874c51a4c83ccc54ebc2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.445474 3312 raft_consensus.cc:348] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.445590 3312 raft_consensus.cc:374] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.445626 3312 raft_consensus.cc:729] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.445753 3311 tablet_bootstrap.cc:654] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.445770 3312 consensus_queue.cc:260] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.446118 3312 ts_tablet_manager.cc:1398] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.446298 3312 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:17.446640 3043 tablet_service.cc:1467] Processing CreateTablet for tablet 71eeb5740a7344749b534273ba1c9922 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.446877 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71eeb5740a7344749b534273ba1c9922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.447542 3312 tablet_bootstrap.cc:654] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.447904 3313 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:17.448763 3312 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:17.448845 3312 ts_tablet_manager.cc:1367] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:17.449303 3312 raft_consensus.cc:348] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.449399 3312 raft_consensus.cc:374] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.449431 3312 raft_consensus.cc:729] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.449438 3313 tablet_bootstrap.cc:654] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.449556 3312 consensus_queue.cc:260] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.449837 3312 ts_tablet_manager.cc:1398] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.449848 2913 tablet_service.cc:1467] Processing CreateTablet for tablet 71eeb5740a7344749b534273ba1c9922 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.450006 3312 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:17.450073 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71eeb5740a7344749b534273ba1c9922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.450902 3312 tablet_bootstrap.cc:654] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.450960 3313 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:17.451036 3313 ts_tablet_manager.cc:1367] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:17.451572 3313 raft_consensus.cc:348] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.451682 3313 raft_consensus.cc:374] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.451725 3313 raft_consensus.cc:729] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.451889 3313 consensus_queue.cc:260] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.452263 3313 ts_tablet_manager.cc:1398] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.452522 3318 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:17.452780 3312 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:17.452855 3312 ts_tablet_manager.cc:1367] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:17.453298 3312 raft_consensus.cc:348] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.453400 3312 raft_consensus.cc:374] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.453430 3312 raft_consensus.cc:729] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.453564 3312 consensus_queue.cc:260] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.453833 3312 ts_tablet_manager.cc:1398] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.454650 3318 tablet_bootstrap.cc:654] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.455744 3311 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:17.455842 3311 ts_tablet_manager.cc:1367] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:17.456415 3311 raft_consensus.cc:348] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.456538 3311 raft_consensus.cc:374] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.456575 3311 raft_consensus.cc:729] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.456699 3311 consensus_queue.cc:260] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.457046 3311 ts_tablet_manager.cc:1398] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:17.457255 3311 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:17.458073 3311 tablet_bootstrap.cc:654] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.457733 2520 tablet_service.cc:1467] Processing CreateTablet for tablet ccac60c83bd54650a50a07d87e3fc72a (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.458264 2389 tablet_service.cc:1467] Processing CreateTablet for tablet ccac60c83bd54650a50a07d87e3fc72a (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.459122 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccac60c83bd54650a50a07d87e3fc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.458643 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccac60c83bd54650a50a07d87e3fc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.458942 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 71eeb5740a7344749b534273ba1c9922 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.459445 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71eeb5740a7344749b534273ba1c9922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.459589 2387 tablet_service.cc:1467] Processing CreateTablet for tablet 71eeb5740a7344749b534273ba1c9922 (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.459837 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71eeb5740a7344749b534273ba1c9922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.460314 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 66e126a09c874c51a4c83ccc54ebc2ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.460691 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e126a09c874c51a4c83ccc54ebc2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.461200 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 66e126a09c874c51a4c83ccc54ebc2ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.461454 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e126a09c874c51a4c83ccc54ebc2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.462639 3313 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:17.463510 3313 tablet_bootstrap.cc:654] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.465873 3311 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:17.465956 3311 ts_tablet_manager.cc:1367] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 07:59:17.466382 3311 raft_consensus.cc:348] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.466516 3311 raft_consensus.cc:374] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.466557 3311 raft_consensus.cc:729] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.466671 3311 consensus_queue.cc:260] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.466768 3318 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:17.466857 3318 ts_tablet_manager.cc:1367] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240503 07:59:17.466921 3311 ts_tablet_manager.cc:1398] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:17.467504 3318 raft_consensus.cc:348] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.467664 3318 raft_consensus.cc:374] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.467729 3318 raft_consensus.cc:729] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.467890 3318 consensus_queue.cc:260] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.468219 3318 ts_tablet_manager.cc:1398] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.468518 3318 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:17.468780 3327 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:17.469121 3326 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:17.469549 3318 tablet_bootstrap.cc:654] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.470891 3313 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:17.470978 3313 ts_tablet_manager.cc:1367] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:17.471292 3318 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:17.471380 3318 ts_tablet_manager.cc:1367] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:17.471401 3326 tablet_bootstrap.cc:654] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.471390 3313 raft_consensus.cc:348] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.471513 3313 raft_consensus.cc:374] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.471555 3313 raft_consensus.cc:729] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.471683 3313 consensus_queue.cc:260] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.471869 3318 raft_consensus.cc:348] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.471968 3313 ts_tablet_manager.cc:1398] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.471983 3318 raft_consensus.cc:374] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.472024 3318 raft_consensus.cc:729] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.472148 3318 consensus_queue.cc:260] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.472415 3327 tablet_bootstrap.cc:654] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.472430 3318 ts_tablet_manager.cc:1398] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.472462 3326 log.cc:824] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:17.473129 3327 log.cc:824] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:17.474583 3326 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:17.474620 3327 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:17.474737 3326 ts_tablet_manager.cc:1367] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 07:59:17.474761 3327 ts_tablet_manager.cc:1367] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 07:59:17.477900 3326 raft_consensus.cc:348] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.478075 3326 raft_consensus.cc:374] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.478111 3326 raft_consensus.cc:729] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.478272 3327 raft_consensus.cc:348] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.478437 3327 raft_consensus.cc:374] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.478502 3327 raft_consensus.cc:729] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.478483 3326 consensus_queue.cc:260] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.478855 3327 consensus_queue.cc:260] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.479274 3326 ts_tablet_manager.cc:1398] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:17.479612 3326 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:17.480387 3326 tablet_bootstrap.cc:654] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.480387 3327 ts_tablet_manager.cc:1398] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 07:59:17.480603 3327 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:17.481391 3326 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:17.481474 3326 ts_tablet_manager.cc:1367] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:17.481551 3327 tablet_bootstrap.cc:654] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.481943 3326 raft_consensus.cc:348] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.482046 3326 raft_consensus.cc:374] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.482077 3326 raft_consensus.cc:729] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.482180 3326 consensus_queue.cc:260] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.482410 3326 ts_tablet_manager.cc:1398] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.482554 3326 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:17.482576 3327 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:17.482637 3327 ts_tablet_manager.cc:1367] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:17.483088 3327 raft_consensus.cc:348] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.483181 3327 raft_consensus.cc:374] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.483211 3327 raft_consensus.cc:729] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.483355 3326 tablet_bootstrap.cc:654] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.483348 3327 consensus_queue.cc:260] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.483673 3327 ts_tablet_manager.cc:1398] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.483821 3327 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:17.484412 3326 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:17.484485 3326 ts_tablet_manager.cc:1367] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:17.484617 3327 tablet_bootstrap.cc:654] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.484903 3326 raft_consensus.cc:348] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.484987 3326 raft_consensus.cc:374] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.485015 3326 raft_consensus.cc:729] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.485141 3326 consensus_queue.cc:260] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.485378 3326 ts_tablet_manager.cc:1398] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.486093 3327 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:17.486181 3327 ts_tablet_manager.cc:1367] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:17.486608 3327 raft_consensus.cc:348] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.486745 3327 raft_consensus.cc:374] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.486791 3327 raft_consensus.cc:729] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.486944 3327 consensus_queue.cc:260] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.487260 3327 ts_tablet_manager.cc:1398] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:17.525861 2587 tablet.cc:2369] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:17.525897 3315 raft_consensus.cc:482] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.526010 3315 raft_consensus.cc:504] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.527303 3315 leader_election.cc:288] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:17.532269 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e126a09c874c51a4c83ccc54ebc2ff" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:17.532421 2802 raft_consensus.cc:2454] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:17.533121 2540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e126a09c874c51a4c83ccc54ebc2ff" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3" is_pre_election: true
I20240503 07:59:17.533393 2540 raft_consensus.cc:2454] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:17.533433 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e126a09c874c51a4c83ccc54ebc2ff" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:17.533623 3062 raft_consensus.cc:2454] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:17.533779 2604 leader_election.cc:302] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 3631cd1b9b79487581e9d31178f671c3, 886e4eb306064b98ab923623b8f1d401; no voters:
I20240503 07:59:17.534013 3315 raft_consensus.cc:2781] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.534080 3315 raft_consensus.cc:482] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.534126 3315 raft_consensus.cc:3037] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.534319 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e126a09c874c51a4c83ccc54ebc2ff" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:17.534562 2409 raft_consensus.cc:2454] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:17.535182 3315 raft_consensus.cc:504] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.535648 3315 leader_election.cc:288] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 election: Requested vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:17.535910 2540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e126a09c874c51a4c83ccc54ebc2ff" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3"
I20240503 07:59:17.535913 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e126a09c874c51a4c83ccc54ebc2ff" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:17.536008 2540 raft_consensus.cc:3037] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.536022 2802 raft_consensus.cc:3037] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.536110 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e126a09c874c51a4c83ccc54ebc2ff" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:17.536197 2409 raft_consensus.cc:3037] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.536790 2540 raft_consensus.cc:2454] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:17.536965 2802 raft_consensus.cc:2454] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:17.537155 2409 raft_consensus.cc:2454] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:17.537364 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e126a09c874c51a4c83ccc54ebc2ff" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:17.537392 2604 leader_election.cc:302] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, 886e4eb306064b98ab923623b8f1d401, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:17.537475 3062 raft_consensus.cc:3037] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.537560 3315 raft_consensus.cc:2781] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.537737 3315 raft_consensus.cc:686] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Becoming Leader. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Running, Role: LEADER
I20240503 07:59:17.538023 3315 consensus_queue.cc:237] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:17.538390 3062 raft_consensus.cc:2454] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:17.540436 2267 catalog_manager.cc:5484] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 886e4eb306064b98ab923623b8f1d401 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:17.616401 2456 tablet.cc:2369] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:17.641803 3331 raft_consensus.cc:482] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.641933 3331 raft_consensus.cc:504] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.643143 3331 leader_election.cc:288] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:17.647608 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71eeb5740a7344749b534273ba1c9922" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:17.647773 3062 raft_consensus.cc:2454] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:17.648123 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71eeb5740a7344749b534273ba1c9922" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:17.648257 2933 raft_consensus.cc:2454] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:17.648468 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71eeb5740a7344749b534273ba1c9922" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:17.648588 2409 raft_consensus.cc:2454] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:17.648850 2473 leader_election.cc:302] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, 81b9495629bb458892b031db80bc3715, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:17.648942 2671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71eeb5740a7344749b534273ba1c9922" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:17.649101 2671 raft_consensus.cc:2454] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:17.649142 3331 raft_consensus.cc:2781] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.649190 3331 raft_consensus.cc:482] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.649222 3331 raft_consensus.cc:3037] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.650019 3331 raft_consensus.cc:504] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.650413 3331 leader_election.cc:288] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Requested vote from peers e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:17.650631 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71eeb5740a7344749b534273ba1c9922" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:17.650648 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71eeb5740a7344749b534273ba1c9922" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:17.650749 2409 raft_consensus.cc:3037] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.650760 2933 raft_consensus.cc:3037] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.650811 2671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71eeb5740a7344749b534273ba1c9922" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:17.650807 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71eeb5740a7344749b534273ba1c9922" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:17.650902 2671 raft_consensus.cc:3037] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.650904 3062 raft_consensus.cc:3037] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.651630 2933 raft_consensus.cc:2454] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:17.651630 2409 raft_consensus.cc:2454] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:17.651710 3062 raft_consensus.cc:2454] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:17.651713 2671 raft_consensus.cc:2454] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:17.652004 2474 leader_election.cc:302] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, 886e4eb306064b98ab923623b8f1d401, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:17.652155 3331 raft_consensus.cc:2781] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.652320 3331 raft_consensus.cc:686] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Running, Role: LEADER
I20240503 07:59:17.652626 3331 consensus_queue.cc:237] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:17.655658 2267 catalog_manager.cc:5484] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:17.749902 3331 raft_consensus.cc:482] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.750121 3331 raft_consensus.cc:504] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.751125 2671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac60c83bd54650a50a07d87e3fc72a" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:17.751125 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac60c83bd54650a50a07d87e3fc72a" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:17.751268 2933 raft_consensus.cc:2454] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:17.751269 2671 raft_consensus.cc:2454] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:17.751271 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac60c83bd54650a50a07d87e3fc72a" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:17.751384 2409 raft_consensus.cc:2454] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:17.751655 2474 leader_election.cc:302] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, 886e4eb306064b98ab923623b8f1d401, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:17.751698 3331 leader_election.cc:288] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:17.751889 3331 raft_consensus.cc:2781] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.751935 3331 raft_consensus.cc:482] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.751969 3331 raft_consensus.cc:3037] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.752961 3331 raft_consensus.cc:504] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.753525 3331 leader_election.cc:288] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:17.753669 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac60c83bd54650a50a07d87e3fc72a" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:17.753756 2409 raft_consensus.cc:3037] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.753839 2671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac60c83bd54650a50a07d87e3fc72a" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:17.753921 2671 raft_consensus.cc:3037] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.753962 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac60c83bd54650a50a07d87e3fc72a" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:17.754065 2933 raft_consensus.cc:3037] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.754621 2409 raft_consensus.cc:2454] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:17.754751 2671 raft_consensus.cc:2454] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:17.754973 2475 leader_election.cc:302] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, 886e4eb306064b98ab923623b8f1d401, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:17.755013 2933 raft_consensus.cc:2454] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:17.755046 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac60c83bd54650a50a07d87e3fc72a" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:17.755079 2801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac60c83bd54650a50a07d87e3fc72a" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:17.755179 2802 raft_consensus.cc:2454] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:17.755345 3351 raft_consensus.cc:2781] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.755537 3351 raft_consensus.cc:686] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Running, Role: LEADER
I20240503 07:59:17.755708 3351 consensus_queue.cc:237] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:17.757422 2267 catalog_manager.cc:5484] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:17.908869 3355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:17.909117 3355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:17.912041 3355 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 07:59:17.983316 3315 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.986460 3062 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.987206 2540 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.987284 2409 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.987787 3343 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.989246 3315 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.989522 3315 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.993270 2671 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.993782 2933 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.993755 3351 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.994196 3351 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.995370 2670 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.995914 2408 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.996673 3331 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.996945 3331 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.997665 2801 raft_consensus.cc:3037] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.998718 2801 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.999768 3351 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.005326 3063 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.005563 2932 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.006111 3331 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.006381 3331 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.009364 3409 mvcc.cc:205] Tried to move back new op lower bound from 7023506055105208320 to 7023506053278613504. Current Snapshot: MvccSnapshot[applied={T|T < 7023506055105208320}]
I20240503 07:59:18.011148 2409 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.011662 3353 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.013872 3419 mvcc.cc:205] Tried to move back new op lower bound from 7023506055105208320 to 7023506053278613504. Current Snapshot: MvccSnapshot[applied={T|T < 7023506055105208320}]
I20240503 07:59:18.019347 3386 mvcc.cc:205] Tried to move back new op lower bound from 7023506055105208320 to 7023506053278613504. Current Snapshot: MvccSnapshot[applied={T|T < 7023506055105208320}]
I20240503 07:59:18.025235 3395 mvcc.cc:205] Tried to move back new op lower bound from 7023506055133028352 to 7023506053748416512. Current Snapshot: MvccSnapshot[applied={T|T < 7023506055133028352}]
I20240503 07:59:18.032954 3413 mvcc.cc:205] Tried to move back new op lower bound from 7023506055135563776 to 7023506054169317376. Current Snapshot: MvccSnapshot[applied={T|T < 7023506055135563776}]
I20240503 07:59:18.035382 3397 mvcc.cc:205] Tried to move back new op lower bound from 7023506055133028352 to 7023506053748416512. Current Snapshot: MvccSnapshot[applied={T|T < 7023506055133028352}]
INSERT report
rows total: 2000
time total: 156.01 ms
time per row: 0.0780048 ms
I20240503 07:59:18.106110 2266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45856:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:18.111233 2781 tablet_service.cc:1467] Processing CreateTablet for tablet dbbc403671d24146b48f8e0c14db7de6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.111300 2389 tablet_service.cc:1467] Processing CreateTablet for tablet dbbc403671d24146b48f8e0c14db7de6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.111569 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbc403671d24146b48f8e0c14db7de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.111567 2781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbc403671d24146b48f8e0c14db7de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.112749 3044 tablet_service.cc:1467] Processing CreateTablet for tablet dbbc403671d24146b48f8e0c14db7de6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.112991 3044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbc403671d24146b48f8e0c14db7de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.113660 3440 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:18.114209 2913 tablet_service.cc:1467] Processing CreateTablet for tablet dbbc403671d24146b48f8e0c14db7de6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.114327 2912 tablet_service.cc:1467] Processing CreateTablet for tablet c9d779a081cd48628e24d1321f7b42af (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.114434 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbc403671d24146b48f8e0c14db7de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.114449 3440 tablet_bootstrap.cc:654] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.114567 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d779a081cd48628e24d1321f7b42af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.115082 3043 tablet_service.cc:1467] Processing CreateTablet for tablet c9d779a081cd48628e24d1321f7b42af (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.115279 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d779a081cd48628e24d1321f7b42af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.116279 3440 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:18.116356 3440 ts_tablet_manager.cc:1367] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:18.116436 2911 tablet_service.cc:1467] Processing CreateTablet for tablet 6a11003820e14dc7b2615f20baf179cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.116650 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a11003820e14dc7b2615f20baf179cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.116777 3440 raft_consensus.cc:348] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.116897 3440 raft_consensus.cc:374] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.116931 3440 raft_consensus.cc:729] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.117111 3442 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:18.117050 3440 consensus_queue.cc:260] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.117337 3440 ts_tablet_manager.cc:1398] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.117859 3444 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:18.118888 3444 tablet_bootstrap.cc:654] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.118923 3442 tablet_bootstrap.cc:654] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.118916 2781 tablet_service.cc:1467] Processing CreateTablet for tablet c9d779a081cd48628e24d1321f7b42af (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.119119 2781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d779a081cd48628e24d1321f7b42af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.120558 2782 tablet_service.cc:1467] Processing CreateTablet for tablet 6a11003820e14dc7b2615f20baf179cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.120718 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a11003820e14dc7b2615f20baf179cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.121342 3441 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:18.121580 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 6a11003820e14dc7b2615f20baf179cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.121788 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a11003820e14dc7b2615f20baf179cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.122026 2520 tablet_service.cc:1467] Processing CreateTablet for tablet c9d779a081cd48628e24d1321f7b42af (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.122164 3441 tablet_bootstrap.cc:654] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.122263 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d779a081cd48628e24d1321f7b42af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.122433 2518 tablet_service.cc:1467] Processing CreateTablet for tablet dbbc403671d24146b48f8e0c14db7de6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.122660 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbc403671d24146b48f8e0c14db7de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.123335 3442 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:18.123404 3442 ts_tablet_manager.cc:1367] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:18.123798 2650 tablet_service.cc:1467] Processing CreateTablet for tablet 6a11003820e14dc7b2615f20baf179cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.123808 3442 raft_consensus.cc:348] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.123903 3442 raft_consensus.cc:374] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.123930 3442 raft_consensus.cc:729] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.124044 2650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a11003820e14dc7b2615f20baf179cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.124348 3444 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:18.124420 3444 ts_tablet_manager.cc:1367] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:18.124408 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 6a11003820e14dc7b2615f20baf179cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.124594 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a11003820e14dc7b2615f20baf179cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.124763 2389 tablet_service.cc:1467] Processing CreateTablet for tablet c9d779a081cd48628e24d1321f7b42af (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.124838 3444 raft_consensus.cc:348] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.124940 3444 raft_consensus.cc:374] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.124958 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d779a081cd48628e24d1321f7b42af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.124970 3444 raft_consensus.cc:729] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.125120 3444 consensus_queue.cc:260] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.125384 3444 ts_tablet_manager.cc:1398] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.125535 3444 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:18.126029 3440 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:18.126307 3444 tablet_bootstrap.cc:654] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.126813 3440 tablet_bootstrap.cc:654] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.126863 3442 consensus_queue.cc:260] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.126994 3448 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:18.127126 3442 ts_tablet_manager.cc:1398] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:18.127269 3442 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:18.127715 3448 tablet_bootstrap.cc:654] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.128141 3441 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:18.128244 3441 ts_tablet_manager.cc:1367] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:18.128409 3442 tablet_bootstrap.cc:654] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.129164 3441 raft_consensus.cc:348] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.129496 3444 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:18.129573 3444 ts_tablet_manager.cc:1367] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:18.129580 3441 raft_consensus.cc:374] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.129622 3441 raft_consensus.cc:729] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.129793 3441 consensus_queue.cc:260] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.129972 3444 raft_consensus.cc:348] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.130044 3441 ts_tablet_manager.cc:1398] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.130065 3444 raft_consensus.cc:374] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.130096 3444 raft_consensus.cc:729] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.130184 3442 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:18.130201 3441 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:18.130194 3444 consensus_queue.cc:260] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.130249 3442 ts_tablet_manager.cc:1367] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:18.130442 3444 ts_tablet_manager.cc:1398] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.130604 3444 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:18.130602 3442 raft_consensus.cc:348] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.130694 3442 raft_consensus.cc:374] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.130720 3442 raft_consensus.cc:729] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.130832 3442 consensus_queue.cc:260] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.130985 3441 tablet_bootstrap.cc:654] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.131072 3442 ts_tablet_manager.cc:1398] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.131361 3444 tablet_bootstrap.cc:654] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.131942 3441 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:18.132011 3441 ts_tablet_manager.cc:1367] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.132403 3441 raft_consensus.cc:348] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.132535 3441 raft_consensus.cc:374] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.132591 3441 raft_consensus.cc:729] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.132720 3441 consensus_queue.cc:260] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.133020 3441 ts_tablet_manager.cc:1398] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.133138 3440 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:18.133195 3441 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:18.133365 3444 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:18.133438 3444 ts_tablet_manager.cc:1367] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:18.133827 3444 raft_consensus.cc:348] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.133919 3444 raft_consensus.cc:374] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.133949 3444 raft_consensus.cc:729] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.134056 3441 tablet_bootstrap.cc:654] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.134047 3444 consensus_queue.cc:260] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.134305 3444 ts_tablet_manager.cc:1398] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.134830 3440 ts_tablet_manager.cc:1367] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:18.135052 3441 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:18.135159 3441 ts_tablet_manager.cc:1367] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:18.135363 3440 raft_consensus.cc:348] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.135470 3440 raft_consensus.cc:374] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.135543 3440 raft_consensus.cc:729] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.135552 3441 raft_consensus.cc:348] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.135643 3441 raft_consensus.cc:374] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.135677 3441 raft_consensus.cc:729] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.135710 3440 consensus_queue.cc:260] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.135787 3441 consensus_queue.cc:260] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.136027 3441 ts_tablet_manager.cc:1398] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.136574 3440 ts_tablet_manager.cc:1398] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.136734 3440 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:18.137826 3440 tablet_bootstrap.cc:654] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.139034 3448 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:18.139034 3440 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:18.139115 3440 ts_tablet_manager.cc:1367] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:18.139117 3448 ts_tablet_manager.cc:1367] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:59:18.139508 3440 raft_consensus.cc:348] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.139524 3448 raft_consensus.cc:348] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.139621 3448 raft_consensus.cc:374] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.139621 3440 raft_consensus.cc:374] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.139658 3448 raft_consensus.cc:729] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.139658 3440 raft_consensus.cc:729] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.139781 3440 consensus_queue.cc:260] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.139781 3448 consensus_queue.cc:260] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.140020 3440 ts_tablet_manager.cc:1398] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.140020 3448 ts_tablet_manager.cc:1398] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.140177 3448 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:18.140714 3452 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:18.141364 3448 tablet_bootstrap.cc:654] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.141706 3452 tablet_bootstrap.cc:654] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.142364 3448 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:18.142493 3448 ts_tablet_manager.cc:1367] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:18.142652 3452 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:18.142728 3452 ts_tablet_manager.cc:1367] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.142864 3448 raft_consensus.cc:348] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.142944 3448 raft_consensus.cc:374] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.142966 3448 raft_consensus.cc:729] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.143067 3448 consensus_queue.cc:260] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.143110 3452 raft_consensus.cc:348] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.143287 3448 ts_tablet_manager.cc:1398] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.143342 3452 raft_consensus.cc:374] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.143399 3452 raft_consensus.cc:729] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.143443 3448 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:18.143571 3452 consensus_queue.cc:260] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.143663 3387 raft_consensus.cc:482] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.143735 3387 raft_consensus.cc:504] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.143975 3452 ts_tablet_manager.cc:1398] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.144851 3448 tablet_bootstrap.cc:654] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.145051 3387 leader_election.cc:288] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:18.145795 3448 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:18.145848 3448 ts_tablet_manager.cc:1367] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.146170 3448 raft_consensus.cc:348] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.146260 3448 raft_consensus.cc:374] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.146291 3448 raft_consensus.cc:729] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.146385 3448 consensus_queue.cc:260] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.146610 3448 ts_tablet_manager.cc:1398] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.150151 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11003820e14dc7b2615f20baf179cc" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:18.150264 2670 raft_consensus.cc:2454] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 0.
I20240503 07:59:18.150449 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11003820e14dc7b2615f20baf179cc" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:18.150561 2409 raft_consensus.cc:2454] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 0.
I20240503 07:59:18.150905 2737 leader_election.cc:302] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 886e4eb306064b98ab923623b8f1d401, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:18.151103 3387 raft_consensus.cc:2781] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.151165 3387 raft_consensus.cc:482] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.151203 3387 raft_consensus.cc:3037] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.151899 3387 raft_consensus.cc:504] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.152266 3387 leader_election.cc:288] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [CANDIDATE]: Term 1 election: Requested vote from peers 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:18.152423 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11003820e14dc7b2615f20baf179cc" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:18.152508 2670 raft_consensus.cc:3037] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.152558 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11003820e14dc7b2615f20baf179cc" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:18.152648 2409 raft_consensus.cc:3037] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.153383 2670 raft_consensus.cc:2454] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 1.
I20240503 07:59:18.153504 2409 raft_consensus.cc:2454] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 1.
I20240503 07:59:18.154127 2735 leader_election.cc:302] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 886e4eb306064b98ab923623b8f1d401, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:18.154412 3387 raft_consensus.cc:2781] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.154560 3387 raft_consensus.cc:686] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Running, Role: LEADER
I20240503 07:59:18.154814 3387 consensus_queue.cc:237] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:18.156682 2932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11003820e14dc7b2615f20baf179cc" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:18.156817 2932 raft_consensus.cc:2454] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 0.
I20240503 07:59:18.157143 2265 catalog_manager.cc:5484] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:18.156723 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11003820e14dc7b2615f20baf179cc" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:18.157686 2933 raft_consensus.cc:3037] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.158514 2933 raft_consensus.cc:2454] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 1.
I20240503 07:59:18.159392 2540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11003820e14dc7b2615f20baf179cc" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3" is_pre_election: true
I20240503 07:59:18.159502 2540 raft_consensus.cc:2454] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 0.
I20240503 07:59:18.159685 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a11003820e14dc7b2615f20baf179cc" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3"
I20240503 07:59:18.159751 2539 raft_consensus.cc:3037] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.160583 2539 raft_consensus.cc:2454] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 1.
I20240503 07:59:18.200435 3293 raft_consensus.cc:482] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.200539 3293 raft_consensus.cc:504] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.201085 3293 leader_election.cc:288] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199)
I20240503 07:59:18.201406 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d779a081cd48628e24d1321f7b42af" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:18.201550 2802 raft_consensus.cc:2454] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:18.201579 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d779a081cd48628e24d1321f7b42af" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:18.201709 2933 raft_consensus.cc:2454] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:18.202000 2998 leader_election.cc:302] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 81b9495629bb458892b031db80bc3715, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:18.202193 3293 raft_consensus.cc:2781] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.202244 3293 raft_consensus.cc:482] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.202280 3293 raft_consensus.cc:3037] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.203272 3293 raft_consensus.cc:504] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.203722 3293 leader_election.cc:288] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 election: Requested vote from peers 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199)
I20240503 07:59:18.204599 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d779a081cd48628e24d1321f7b42af" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:18.204697 2933 raft_consensus.cc:3037] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.204946 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d779a081cd48628e24d1321f7b42af" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:18.205030 2802 raft_consensus.cc:3037] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.205605 2933 raft_consensus.cc:2454] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:18.205780 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d779a081cd48628e24d1321f7b42af" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:18.205884 2802 raft_consensus.cc:2454] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:18.205919 2409 raft_consensus.cc:2454] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:18.205974 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d779a081cd48628e24d1321f7b42af" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:18.206077 2408 raft_consensus.cc:3037] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.206422 3000 leader_election.cc:302] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 81b9495629bb458892b031db80bc3715, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:18.206638 3293 raft_consensus.cc:2781] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.206815 3293 raft_consensus.cc:686] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Becoming Leader. State: Replica: 81b9495629bb458892b031db80bc3715, State: Running, Role: LEADER
I20240503 07:59:18.206921 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d779a081cd48628e24d1321f7b42af" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3" is_pre_election: true
I20240503 07:59:18.206996 2408 raft_consensus.cc:2454] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:18.206955 3293 consensus_queue.cc:237] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.207033 2539 raft_consensus.cc:2454] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:18.207191 2540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d779a081cd48628e24d1321f7b42af" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3"
I20240503 07:59:18.207253 2540 raft_consensus.cc:3037] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.208113 2540 raft_consensus.cc:2454] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:18.209157 2265 catalog_manager.cc:5484] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81b9495629bb458892b031db80bc3715 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.237372 3438 raft_consensus.cc:482] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.237466 3438 raft_consensus.cc:504] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.237885 3438 leader_election.cc:288] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:18.238107 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbc403671d24146b48f8e0c14db7de6" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:18.238232 2802 raft_consensus.cc:2454] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:18.238245 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbc403671d24146b48f8e0c14db7de6" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:18.238231 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbc403671d24146b48f8e0c14db7de6" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:18.238343 2933 raft_consensus.cc:2454] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:18.238354 2408 raft_consensus.cc:2454] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:18.238338 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbc403671d24146b48f8e0c14db7de6" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:18.238435 3063 raft_consensus.cc:2454] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:18.238652 2476 leader_election.cc:302] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 3631cd1b9b79487581e9d31178f671c3, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:18.238812 3438 raft_consensus.cc:2781] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.238867 3438 raft_consensus.cc:482] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.238931 3438 raft_consensus.cc:3037] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.239712 3438 raft_consensus.cc:504] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.240103 3438 leader_election.cc:288] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:18.240306 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbc403671d24146b48f8e0c14db7de6" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:18.240319 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbc403671d24146b48f8e0c14db7de6" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:18.240406 2408 raft_consensus.cc:3037] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.240406 2933 raft_consensus.cc:3037] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.241107 2933 raft_consensus.cc:2454] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:18.241114 2408 raft_consensus.cc:2454] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:18.241293 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbc403671d24146b48f8e0c14db7de6" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:18.241392 2802 raft_consensus.cc:3037] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.241387 2474 leader_election.cc:302] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, cd82dbc60d0f40688fbe8e712dc7107a, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:18.241473 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbc403671d24146b48f8e0c14db7de6" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:18.241524 3438 raft_consensus.cc:2781] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.241567 3063 raft_consensus.cc:3037] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.241598 3438 raft_consensus.cc:686] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Running, Role: LEADER
I20240503 07:59:18.241714 3438 consensus_queue.cc:237] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.242250 2802 raft_consensus.cc:2454] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:18.242398 3063 raft_consensus.cc:2454] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:18.243317 2265 catalog_manager.cc:5484] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:18.365206 3471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:18.365411 3471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:18.368469 3471 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 07:59:18.439502 2933 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.440073 3468 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.443465 2540 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.443843 2670 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.444305 2408 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.444795 3387 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.444869 2802 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.445237 3387 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.445297 2932 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.445443 3465 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.445927 3465 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.447433 2408 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.448633 3063 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.448773 3468 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.450735 2539 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.451587 3293 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.452132 2802 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.452790 2408 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.454264 3331 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.454852 3353 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.456055 3353 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.459110 2931 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.459542 3293 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.460348 3503 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 121.794 ms
time per row: 0.060897 ms
I20240503 07:59:18.540766 2264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45856:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:18.547219 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.547219 2389 tablet_service.cc:1467] Processing CreateTablet for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.547482 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.547482 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.548899 2782 tablet_service.cc:1467] Processing CreateTablet for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.548995 2912 tablet_service.cc:1467] Processing CreateTablet for tablet df62f8897f4f4e4abe13f3455a665031 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.549257 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df62f8897f4f4e4abe13f3455a665031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.549283 2781 tablet_service.cc:1467] Processing CreateTablet for tablet df62f8897f4f4e4abe13f3455a665031 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.549494 2781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df62f8897f4f4e4abe13f3455a665031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.550881 3441 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:18.551054 3444 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:18.551090 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 7d732123aab240c4b4216f451e87de09 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.551293 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d732123aab240c4b4216f451e87de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.551604 3441 tablet_bootstrap.cc:654] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.551777 3444 tablet_bootstrap.cc:654] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.552930 2651 tablet_service.cc:1467] Processing CreateTablet for tablet 7d732123aab240c4b4216f451e87de09 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.552938 2649 tablet_service.cc:1467] Processing CreateTablet for tablet df62f8897f4f4e4abe13f3455a665031 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.553229 2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d732123aab240c4b4216f451e87de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.553205 2650 tablet_service.cc:1467] Processing CreateTablet for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.553333 2649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df62f8897f4f4e4abe13f3455a665031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.553388 2911 tablet_service.cc:1467] Processing CreateTablet for tablet 7d732123aab240c4b4216f451e87de09 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.553586 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d732123aab240c4b4216f451e87de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.553606 3043 tablet_service.cc:1467] Processing CreateTablet for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.553845 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.555214 3448 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:18.555697 3442 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:18.555763 3440 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:18.555881 3448 tablet_bootstrap.cc:654] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.556453 3442 tablet_bootstrap.cc:654] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.556499 3440 tablet_bootstrap.cc:654] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.556586 2387 tablet_service.cc:1467] Processing CreateTablet for tablet df62f8897f4f4e4abe13f3455a665031 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.556780 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df62f8897f4f4e4abe13f3455a665031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.557518 3044 tablet_service.cc:1467] Processing CreateTablet for tablet 7d732123aab240c4b4216f451e87de09 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.557708 3044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d732123aab240c4b4216f451e87de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.558614 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.558732 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 7d732123aab240c4b4216f451e87de09 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.558931 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d732123aab240c4b4216f451e87de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.559463 3042 tablet_service.cc:1467] Processing CreateTablet for tablet df62f8897f4f4e4abe13f3455a665031 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.559654 3042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df62f8897f4f4e4abe13f3455a665031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.561339 3452 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:18.562069 3452 tablet_bootstrap.cc:654] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.562837 3448 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:18.562897 3448 ts_tablet_manager.cc:1367] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:18.563223 3448 raft_consensus.cc:348] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.563311 3448 raft_consensus.cc:374] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.563339 3448 raft_consensus.cc:729] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.563446 3448 consensus_queue.cc:260] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.563475 2650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.563678 3448 ts_tablet_manager.cc:1398] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.563781 3448 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:18.564425 3448 tablet_bootstrap.cc:654] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.565629 3452 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:18.565704 3452 ts_tablet_manager.cc:1367] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:18.566068 3441 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:18.566053 3452 raft_consensus.cc:348] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.566150 3452 raft_consensus.cc:374] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.566149 3441 ts_tablet_manager.cc:1367] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20240503 07:59:18.566177 3452 raft_consensus.cc:729] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.566279 3452 consensus_queue.cc:260] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.566516 3452 ts_tablet_manager.cc:1398] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.566565 3441 raft_consensus.cc:348] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.566660 3441 raft_consensus.cc:374] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.566668 3452 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:18.566687 3441 raft_consensus.cc:729] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.566817 3444 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:18.566790 3441 consensus_queue.cc:260] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.566887 3444 ts_tablet_manager.cc:1367] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:18.567034 3441 ts_tablet_manager.cc:1398] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.567163 3441 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:18.567278 3444 raft_consensus.cc:348] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.567351 3448 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:18.567378 3444 raft_consensus.cc:374] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.567409 3444 raft_consensus.cc:729] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.567421 3452 tablet_bootstrap.cc:654] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.567440 3448 ts_tablet_manager.cc:1367] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:18.567569 3444 consensus_queue.cc:260] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.567826 3444 ts_tablet_manager.cc:1398] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.567845 3448 raft_consensus.cc:348] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.567942 3448 raft_consensus.cc:374] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.567955 3444 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:18.567972 3448 raft_consensus.cc:729] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.567977 3441 tablet_bootstrap.cc:654] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.568739 3444 tablet_bootstrap.cc:654] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.568907 3442 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:18.568991 3442 ts_tablet_manager.cc:1367] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 07:59:18.569424 3442 raft_consensus.cc:348] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.569528 3442 raft_consensus.cc:374] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.569571 3442 raft_consensus.cc:729] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.569658 3440 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:18.569682 3442 consensus_queue.cc:260] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.569739 3440 ts_tablet_manager.cc:1367] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:18.569943 3442 ts_tablet_manager.cc:1398] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.570102 3442 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:18.570159 3440 raft_consensus.cc:348] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.570273 3440 raft_consensus.cc:374] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.570308 3440 raft_consensus.cc:729] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.570423 3440 consensus_queue.cc:260] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.570710 3440 ts_tablet_manager.cc:1398] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.570869 3440 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:18.570919 3442 tablet_bootstrap.cc:654] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.571676 3440 tablet_bootstrap.cc:654] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.571789 3441 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:18.571859 3441 ts_tablet_manager.cc:1367] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 07:59:18.572008 3452 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:18.572077 3452 ts_tablet_manager.cc:1367] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:18.572256 3448 consensus_queue.cc:260] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.572281 3441 raft_consensus.cc:348] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.572362 3441 raft_consensus.cc:374] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.572389 3441 raft_consensus.cc:729] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.572446 3452 raft_consensus.cc:348] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.572489 3441 consensus_queue.cc:260] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.572546 3452 raft_consensus.cc:374] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.572578 3452 raft_consensus.cc:729] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.572690 3452 consensus_queue.cc:260] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.572755 3441 ts_tablet_manager.cc:1398] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.572970 3452 ts_tablet_manager.cc:1398] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.573150 3452 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:18.573364 3440 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:18.573439 3440 ts_tablet_manager.cc:1367] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:18.573894 3452 tablet_bootstrap.cc:654] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.573830 3440 raft_consensus.cc:348] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.573948 3440 raft_consensus.cc:374] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.573982 3440 raft_consensus.cc:729] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.574110 3440 consensus_queue.cc:260] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.574373 3440 ts_tablet_manager.cc:1398] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.574558 3440 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:18.575263 3452 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:18.575331 3440 tablet_bootstrap.cc:654] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.575423 3452 ts_tablet_manager.cc:1367] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:18.575819 3452 raft_consensus.cc:348] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.575901 3452 raft_consensus.cc:374] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.575929 3452 raft_consensus.cc:729] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.576033 3452 consensus_queue.cc:260] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.576323 3452 ts_tablet_manager.cc:1398] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.578225 3448 ts_tablet_manager.cc:1398] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 07:59:18.578866 3440 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:18.578984 3440 ts_tablet_manager.cc:1367] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:18.579061 3444 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:18.579146 3444 ts_tablet_manager.cc:1367] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20240503 07:59:18.579541 3444 raft_consensus.cc:348] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.579672 3444 raft_consensus.cc:374] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.579715 3444 raft_consensus.cc:729] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.579835 3444 consensus_queue.cc:260] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.580103 3444 ts_tablet_manager.cc:1398] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.580425 3440 raft_consensus.cc:348] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.580526 3440 raft_consensus.cc:374] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.580554 3440 raft_consensus.cc:729] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.580675 3440 consensus_queue.cc:260] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.580987 3440 ts_tablet_manager.cc:1398] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:18.581566 3442 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:18.581656 3442 ts_tablet_manager.cc:1367] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:59:18.582031 3442 raft_consensus.cc:348] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.582134 3442 raft_consensus.cc:374] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.582159 3442 raft_consensus.cc:729] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.582260 3442 consensus_queue.cc:260] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.582491 3442 ts_tablet_manager.cc:1398] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.582635 3442 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:18.583492 3442 tablet_bootstrap.cc:654] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.584316 3442 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:18.584378 3442 ts_tablet_manager.cc:1367] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.584721 3442 raft_consensus.cc:348] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.584839 3442 raft_consensus.cc:374] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.584880 3442 raft_consensus.cc:729] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.584988 3442 consensus_queue.cc:260] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.585294 3442 ts_tablet_manager.cc:1398] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.586542 3390 raft_consensus.cc:482] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.586618 3390 raft_consensus.cc:504] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.586992 3390 leader_election.cc:288] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199)
I20240503 07:59:18.587226 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d4dc031bac4eb1a8fa5d25ca7140b3" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:18.587244 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d4dc031bac4eb1a8fa5d25ca7140b3" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3" is_pre_election: true
I20240503 07:59:18.587306 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d4dc031bac4eb1a8fa5d25ca7140b3" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:18.587343 2407 raft_consensus.cc:2454] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:18.587343 2539 raft_consensus.cc:2454] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:18.587405 3062 raft_consensus.cc:2454] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:18.587415 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d4dc031bac4eb1a8fa5d25ca7140b3" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:18.587536 2802 raft_consensus.cc:2454] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:18.587680 2606 leader_election.cc:302] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, 81b9495629bb458892b031db80bc3715, 886e4eb306064b98ab923623b8f1d401; no voters:
I20240503 07:59:18.587921 3390 raft_consensus.cc:2781] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.587967 3390 raft_consensus.cc:482] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.588001 3390 raft_consensus.cc:3037] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.588732 3390 raft_consensus.cc:504] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.589165 3390 leader_election.cc:288] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 election: Requested vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199)
I20240503 07:59:18.589416 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d4dc031bac4eb1a8fa5d25ca7140b3" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:18.589465 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d4dc031bac4eb1a8fa5d25ca7140b3" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:18.589493 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d4dc031bac4eb1a8fa5d25ca7140b3" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:18.589524 3062 raft_consensus.cc:3037] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.589525 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d4dc031bac4eb1a8fa5d25ca7140b3" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3"
I20240503 07:59:18.589565 2802 raft_consensus.cc:3037] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.589586 2407 raft_consensus.cc:3037] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.589602 2539 raft_consensus.cc:3037] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.590377 3062 raft_consensus.cc:2454] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:18.590377 2802 raft_consensus.cc:2454] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:18.590473 2407 raft_consensus.cc:2454] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:18.590473 2539 raft_consensus.cc:2454] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:18.590792 2607 leader_election.cc:302] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 81b9495629bb458892b031db80bc3715, 886e4eb306064b98ab923623b8f1d401; no voters:
I20240503 07:59:18.591112 3390 raft_consensus.cc:2781] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.591259 3390 raft_consensus.cc:686] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Becoming Leader. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Running, Role: LEADER
I20240503 07:59:18.591467 3390 consensus_queue.cc:237] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } }
I20240503 07:59:18.592929 2263 catalog_manager.cc:5484] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 886e4eb306064b98ab923623b8f1d401 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:18.631429 3515 raft_consensus.cc:482] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.631527 3515 raft_consensus.cc:504] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.632022 3515 leader_election.cc:288] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251)
I20240503 07:59:18.632241 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df62f8897f4f4e4abe13f3455a665031" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:18.632236 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df62f8897f4f4e4abe13f3455a665031" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:18.632366 2670 raft_consensus.cc:2454] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:18.632370 2802 raft_consensus.cc:2454] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:18.632362 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df62f8897f4f4e4abe13f3455a665031" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:18.632474 2407 raft_consensus.cc:2454] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:18.632583 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df62f8897f4f4e4abe13f3455a665031" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:18.632711 2933 raft_consensus.cc:2454] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 0.
I20240503 07:59:18.632745 2999 leader_election.cc:302] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 81b9495629bb458892b031db80bc3715, 886e4eb306064b98ab923623b8f1d401; no voters:
I20240503 07:59:18.632899 3515 raft_consensus.cc:2781] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.632951 3515 raft_consensus.cc:482] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.632987 3515 raft_consensus.cc:3037] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.633837 3515 raft_consensus.cc:504] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.634253 3515 leader_election.cc:288] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 election: Requested vote from peers 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251)
I20240503 07:59:18.634438 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df62f8897f4f4e4abe13f3455a665031" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:18.634483 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df62f8897f4f4e4abe13f3455a665031" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:18.634557 2670 raft_consensus.cc:3037] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.634557 2802 raft_consensus.cc:3037] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.635239 2802 raft_consensus.cc:2454] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:18.635566 2670 raft_consensus.cc:2454] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:18.635811 2999 leader_election.cc:302] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 81b9495629bb458892b031db80bc3715, 886e4eb306064b98ab923623b8f1d401; no voters:
I20240503 07:59:18.635942 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df62f8897f4f4e4abe13f3455a665031" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:18.636039 2407 raft_consensus.cc:3037] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.636065 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df62f8897f4f4e4abe13f3455a665031" candidate_uuid: "81b9495629bb458892b031db80bc3715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:18.636183 2933 raft_consensus.cc:3037] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.636715 2407 raft_consensus.cc:2454] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:18.636883 3515 raft_consensus.cc:2781] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.636965 3515 raft_consensus.cc:686] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Becoming Leader. State: Replica: 81b9495629bb458892b031db80bc3715, State: Running, Role: LEADER
I20240503 07:59:18.637125 3515 consensus_queue.cc:237] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } }
I20240503 07:59:18.637143 2933 raft_consensus.cc:2454] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81b9495629bb458892b031db80bc3715 in term 1.
I20240503 07:59:18.638640 2263 catalog_manager.cc:5484] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81b9495629bb458892b031db80bc3715 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:18.746891 3353 raft_consensus.cc:482] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.747025 3353 raft_consensus.cc:504] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.747509 3353 leader_election.cc:288] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:18.747818 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d732123aab240c4b4216f451e87de09" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:18.747829 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d732123aab240c4b4216f451e87de09" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:18.747918 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d732123aab240c4b4216f451e87de09" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:18.747933 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d732123aab240c4b4216f451e87de09" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:18.747962 2670 raft_consensus.cc:2454] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:18.748054 3062 raft_consensus.cc:2454] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:18.748054 2933 raft_consensus.cc:2454] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:18.748064 2407 raft_consensus.cc:2454] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:18.748303 2473 leader_election.cc:302] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, 886e4eb306064b98ab923623b8f1d401, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:18.748555 3353 raft_consensus.cc:2781] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.748616 3353 raft_consensus.cc:482] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.748646 3353 raft_consensus.cc:3037] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.749406 3353 raft_consensus.cc:504] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.749785 3353 leader_election.cc:288] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Requested vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:18.749991 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d732123aab240c4b4216f451e87de09" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:18.750064 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d732123aab240c4b4216f451e87de09" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:18.750080 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d732123aab240c4b4216f451e87de09" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:18.750144 2407 raft_consensus.cc:3037] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.750161 2670 raft_consensus.cc:3037] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.750166 3062 raft_consensus.cc:3037] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.750216 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d732123aab240c4b4216f451e87de09" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:18.750296 2933 raft_consensus.cc:3037] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.751003 2670 raft_consensus.cc:2454] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:18.751041 3062 raft_consensus.cc:2454] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:18.751041 2407 raft_consensus.cc:2454] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:18.751068 2933 raft_consensus.cc:2454] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:18.751312 2474 leader_election.cc:302] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, cd82dbc60d0f40688fbe8e712dc7107a, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:18.751458 3353 raft_consensus.cc:2781] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.751571 3353 raft_consensus.cc:686] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Running, Role: LEADER
I20240503 07:59:18.751718 3353 consensus_queue.cc:237] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:18.753376 2267 catalog_manager.cc:5484] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:18.897615 3536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:18.897805 3536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:18.900851 3536 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 07:59:18.973770 2407 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.974123 3062 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.974757 2670 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.974833 3353 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.975643 3503 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.976136 2933 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.979732 2802 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.980113 3390 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.980557 2539 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.980564 2409 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.981019 3532 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.981217 2409 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.981323 3532 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.980993 3351 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.981699 3293 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.981861 3353 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.982412 2801 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.982739 3063 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.983265 2932 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.984467 2671 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.986076 3571 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 07:59:18.986374 3468 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.992954 3468 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.001114 3569 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:19.025504 2998 consensus_peers.cc:485] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38745: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
INSERT report
rows total: 2000
time total: 150.85 ms
time per row: 0.0754252 ms
I20240503 07:59:19.090533 2263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45856:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:19.095407 2519 tablet_service.cc:1467] Processing CreateTablet for tablet 813f904dbfab4d46964bbbf877715cc9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.095721 2519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f904dbfab4d46964bbbf877715cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.096378 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4acef14f154d0b8fb94428a45a380b (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.096581 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4acef14f154d0b8fb94428a45a380b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.097218 2912 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4acef14f154d0b8fb94428a45a380b (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.097218 2780 tablet_service.cc:1467] Processing CreateTablet for tablet 401e87d2d48e4515b725c10757178536 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.097455 2912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4acef14f154d0b8fb94428a45a380b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.097458 2780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401e87d2d48e4515b725c10757178536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.097949 2782 tablet_service.cc:1467] Processing CreateTablet for tablet 813f904dbfab4d46964bbbf877715cc9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.098162 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f904dbfab4d46964bbbf877715cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.099581 2913 tablet_service.cc:1467] Processing CreateTablet for tablet 401e87d2d48e4515b725c10757178536 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.099777 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401e87d2d48e4515b725c10757178536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.099772 2389 tablet_service.cc:1467] Processing CreateTablet for tablet 401e87d2d48e4515b725c10757178536 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.099790 2781 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4acef14f154d0b8fb94428a45a380b (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.099992 2781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4acef14f154d0b8fb94428a45a380b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.100006 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401e87d2d48e4515b725c10757178536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.101512 2911 tablet_service.cc:1467] Processing CreateTablet for tablet 813f904dbfab4d46964bbbf877715cc9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.101711 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f904dbfab4d46964bbbf877715cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.101972 3596 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:19.102684 3042 tablet_service.cc:1467] Processing CreateTablet for tablet 813f904dbfab4d46964bbbf877715cc9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.102771 3596 tablet_bootstrap.cc:654] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.102852 2518 tablet_service.cc:1467] Processing CreateTablet for tablet 401e87d2d48e4515b725c10757178536 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.102924 3042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f904dbfab4d46964bbbf877715cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.103047 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401e87d2d48e4515b725c10757178536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.103780 3593 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:19.104468 3593 tablet_bootstrap.cc:654] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.104954 3044 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4acef14f154d0b8fb94428a45a380b (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.105195 3044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4acef14f154d0b8fb94428a45a380b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.105378 3593 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:19.105468 3593 ts_tablet_manager.cc:1367] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.105832 3593 raft_consensus.cc:348] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.105957 3593 raft_consensus.cc:374] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.106000 3593 raft_consensus.cc:729] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.106125 3593 consensus_queue.cc:260] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.106400 3593 ts_tablet_manager.cc:1398] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.106563 3593 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:19.107270 3593 tablet_bootstrap.cc:654] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.107270 2650 tablet_service.cc:1467] Processing CreateTablet for tablet 813f904dbfab4d46964bbbf877715cc9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.107381 2649 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4acef14f154d0b8fb94428a45a380b (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.107558 2650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f904dbfab4d46964bbbf877715cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.107805 2649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4acef14f154d0b8fb94428a45a380b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.108242 3593 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:19.108307 3593 ts_tablet_manager.cc:1367] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.108675 3593 raft_consensus.cc:348] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.108778 3593 raft_consensus.cc:374] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.108806 3593 raft_consensus.cc:729] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.108906 3593 consensus_queue.cc:260] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.109241 3593 ts_tablet_manager.cc:1398] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.109386 3593 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:19.109362 2651 tablet_service.cc:1467] Processing CreateTablet for tablet 401e87d2d48e4515b725c10757178536 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.109553 2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401e87d2d48e4515b725c10757178536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.110028 3598 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:19.110136 3593 tablet_bootstrap.cc:654] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.110360 3601 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:19.110841 3598 tablet_bootstrap.cc:654] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.111071 3593 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:19.111143 3593 ts_tablet_manager.cc:1367] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:19.111306 3601 tablet_bootstrap.cc:654] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.111696 3594 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:19.111644 3593 raft_consensus.cc:348] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.111734 3593 raft_consensus.cc:374] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.111760 3593 raft_consensus.cc:729] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.111856 3593 consensus_queue.cc:260] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.112104 3593 ts_tablet_manager.cc:1398] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.112195 3601 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:19.112267 3601 ts_tablet_manager.cc:1367] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:19.112484 3594 tablet_bootstrap.cc:654] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.112694 3601 raft_consensus.cc:348] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.112813 3601 raft_consensus.cc:374] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.112846 3601 raft_consensus.cc:729] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.112960 3595 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:19.112964 3601 consensus_queue.cc:260] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.113242 3601 ts_tablet_manager.cc:1398] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.113492 3601 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:19.113746 3595 tablet_bootstrap.cc:654] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.114291 3601 tablet_bootstrap.cc:654] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.114444 3598 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:19.114500 3598 ts_tablet_manager.cc:1367] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:19.114862 3598 raft_consensus.cc:348] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.114957 3598 raft_consensus.cc:374] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.114987 3598 raft_consensus.cc:729] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.115082 3598 consensus_queue.cc:260] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.115329 3598 ts_tablet_manager.cc:1398] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.115717 3594 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:19.115760 3601 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:19.115835 3601 ts_tablet_manager.cc:1367] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:19.115837 3594 ts_tablet_manager.cc:1367] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:19.116230 3594 raft_consensus.cc:348] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.116307 3594 raft_consensus.cc:374] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.116325 3594 raft_consensus.cc:729] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.116530 3598 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:19.116891 3601 raft_consensus.cc:348] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.117048 3601 raft_consensus.cc:374] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.117182 3601 raft_consensus.cc:729] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.117360 3601 consensus_queue.cc:260] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.117527 3596 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:19.117599 3598 tablet_bootstrap.cc:654] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.117725 3594 consensus_queue.cc:260] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.118100 3594 ts_tablet_manager.cc:1398] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.118310 3594 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:19.118321 3601 ts_tablet_manager.cc:1398] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.118468 3601 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:19.119143 3594 tablet_bootstrap.cc:654] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.119268 3601 tablet_bootstrap.cc:654] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.119869 3596 ts_tablet_manager.cc:1367] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 07:59:19.120250 3601 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:19.120324 3601 ts_tablet_manager.cc:1367] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:19.120527 3594 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:19.120595 3594 ts_tablet_manager.cc:1367] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.120720 3601 raft_consensus.cc:348] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.120801 3601 raft_consensus.cc:374] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.120829 3601 raft_consensus.cc:729] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.120939 3601 consensus_queue.cc:260] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.120977 3594 raft_consensus.cc:348] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.121096 3594 raft_consensus.cc:374] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.121129 3594 raft_consensus.cc:729] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.121210 3601 ts_tablet_manager.cc:1398] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.121246 3594 consensus_queue.cc:260] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.121498 3594 ts_tablet_manager.cc:1398] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.121748 3594 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:19.122151 3598 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:19.122228 3598 ts_tablet_manager.cc:1367] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:19.122601 3598 raft_consensus.cc:348] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.122687 3594 tablet_bootstrap.cc:654] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.122718 3598 raft_consensus.cc:374] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.122753 3598 raft_consensus.cc:729] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.122953 3598 consensus_queue.cc:260] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.123234 3598 ts_tablet_manager.cc:1398] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:19.120641 3596 raft_consensus.cc:348] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.123692 3596 raft_consensus.cc:374] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.123742 3596 raft_consensus.cc:729] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.123893 3596 consensus_queue.cc:260] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.124117 3595 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:19.124202 3595 ts_tablet_manager.cc:1367] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:19.124217 3596 ts_tablet_manager.cc:1398] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:19.124567 3595 raft_consensus.cc:348] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.124677 3595 raft_consensus.cc:374] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.124709 3595 raft_consensus.cc:729] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.124821 3595 consensus_queue.cc:260] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.125113 3595 ts_tablet_manager.cc:1398] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.125422 3595 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:19.125625 3594 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:19.125756 3594 ts_tablet_manager.cc:1367] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:19.126128 3594 raft_consensus.cc:348] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.126219 3595 tablet_bootstrap.cc:654] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.126233 3594 raft_consensus.cc:374] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.126262 3594 raft_consensus.cc:729] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.126401 3594 consensus_queue.cc:260] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.126677 3594 ts_tablet_manager.cc:1398] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.127867 3595 tablet_bootstrap.cc:492] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:19.127943 3595 ts_tablet_manager.cc:1367] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:19.128330 3595 raft_consensus.cc:348] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.128417 3595 raft_consensus.cc:374] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.128444 3595 raft_consensus.cc:729] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.128544 3595 consensus_queue.cc:260] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.128787 3595 ts_tablet_manager.cc:1398] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.128928 3595 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:19.129758 3595 tablet_bootstrap.cc:654] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.130612 3595 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:19.130697 3595 ts_tablet_manager.cc:1367] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.131147 3595 raft_consensus.cc:348] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.131223 3595 raft_consensus.cc:374] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.131253 3595 raft_consensus.cc:729] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.131356 3595 consensus_queue.cc:260] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.131557 3595 ts_tablet_manager.cc:1398] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.200505 3353 raft_consensus.cc:482] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.200665 3353 raft_consensus.cc:504] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.200726 3580 raft_consensus.cc:482] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.200842 3580 raft_consensus.cc:504] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.201229 3353 leader_election.cc:288] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:19.201520 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401e87d2d48e4515b725c10757178536" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:19.201556 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401e87d2d48e4515b725c10757178536" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:19.201650 2802 raft_consensus.cc:2454] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:19.201686 2409 raft_consensus.cc:2454] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:19.201879 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401e87d2d48e4515b725c10757178536" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:19.201989 2933 raft_consensus.cc:2454] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:19.201977 2473 leader_election.cc:302] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 3631cd1b9b79487581e9d31178f671c3, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:19.202100 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401e87d2d48e4515b725c10757178536" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:19.202147 3353 raft_consensus.cc:2781] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.202190 3353 raft_consensus.cc:482] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.202221 3353 raft_consensus.cc:3037] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.202205 3580 leader_election.cc:288] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:19.202229 2670 raft_consensus.cc:2454] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:19.203274 3353 raft_consensus.cc:504] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.203709 3353 leader_election.cc:288] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Requested vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:19.204428 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401e87d2d48e4515b725c10757178536" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:19.204442 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401e87d2d48e4515b725c10757178536" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:19.204520 2933 raft_consensus.cc:3037] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.204520 2802 raft_consensus.cc:3037] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.205381 2802 raft_consensus.cc:2454] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:19.205381 2933 raft_consensus.cc:2454] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:19.205652 2476 leader_election.cc:302] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 3631cd1b9b79487581e9d31178f671c3, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:19.206068 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401e87d2d48e4515b725c10757178536" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:19.206161 2670 raft_consensus.cc:3037] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.206281 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401e87d2d48e4515b725c10757178536" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:19.206377 2409 raft_consensus.cc:3037] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.207048 2670 raft_consensus.cc:2454] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:19.207150 3353 raft_consensus.cc:2781] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.207230 3353 raft_consensus.cc:686] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Running, Role: LEADER
I20240503 07:59:19.207257 2409 raft_consensus.cc:2454] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:19.207360 3353 consensus_queue.cc:237] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } }
I20240503 07:59:19.208477 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f904dbfab4d46964bbbf877715cc9" candidate_uuid: "e3b61927599645d4b11278833b2f27d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:19.208582 2802 raft_consensus.cc:2454] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3b61927599645d4b11278833b2f27d6 in term 0.
I20240503 07:59:19.209794 2263 catalog_manager.cc:5484] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:19.211302 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f904dbfab4d46964bbbf877715cc9" candidate_uuid: "e3b61927599645d4b11278833b2f27d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3" is_pre_election: true
I20240503 07:59:19.211405 2539 raft_consensus.cc:2454] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3b61927599645d4b11278833b2f27d6 in term 0.
I20240503 07:59:19.211864 2866 leader_election.cc:302] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 3631cd1b9b79487581e9d31178f671c3, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:19.212105 3580 raft_consensus.cc:2781] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.212146 3580 raft_consensus.cc:482] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.212177 3580 raft_consensus.cc:3037] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.213016 3580 raft_consensus.cc:504] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.213387 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f904dbfab4d46964bbbf877715cc9" candidate_uuid: "e3b61927599645d4b11278833b2f27d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:19.213449 3580 leader_election.cc:288] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [CANDIDATE]: Term 1 election: Requested vote from peers 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:19.213491 2670 raft_consensus.cc:2454] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3b61927599645d4b11278833b2f27d6 in term 0.
I20240503 07:59:19.213804 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f904dbfab4d46964bbbf877715cc9" candidate_uuid: "e3b61927599645d4b11278833b2f27d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:19.213888 2670 raft_consensus.cc:3037] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.213879 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f904dbfab4d46964bbbf877715cc9" candidate_uuid: "e3b61927599645d4b11278833b2f27d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:19.213986 3062 raft_consensus.cc:2454] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3b61927599645d4b11278833b2f27d6 in term 0.
I20240503 07:59:19.214152 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f904dbfab4d46964bbbf877715cc9" candidate_uuid: "e3b61927599645d4b11278833b2f27d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:19.214229 3063 raft_consensus.cc:3037] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.214628 2540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f904dbfab4d46964bbbf877715cc9" candidate_uuid: "e3b61927599645d4b11278833b2f27d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3"
I20240503 07:59:19.214697 2670 raft_consensus.cc:2454] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3b61927599645d4b11278833b2f27d6 in term 1.
I20240503 07:59:19.214712 2540 raft_consensus.cc:3037] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.215067 3063 raft_consensus.cc:2454] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3b61927599645d4b11278833b2f27d6 in term 1.
I20240503 07:59:19.215482 2540 raft_consensus.cc:2454] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3b61927599645d4b11278833b2f27d6 in term 1.
I20240503 07:59:19.215500 2868 leader_election.cc:302] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 81b9495629bb458892b031db80bc3715, 886e4eb306064b98ab923623b8f1d401, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:19.215617 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813f904dbfab4d46964bbbf877715cc9" candidate_uuid: "e3b61927599645d4b11278833b2f27d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:19.215701 3580 raft_consensus.cc:2781] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.215721 2802 raft_consensus.cc:3037] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.215778 3580 raft_consensus.cc:686] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 1 LEADER]: Becoming Leader. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Running, Role: LEADER
I20240503 07:59:19.216053 3580 consensus_queue.cc:237] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.216531 2802 raft_consensus.cc:2454] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3b61927599645d4b11278833b2f27d6 in term 1.
I20240503 07:59:19.218062 3587 raft_consensus.cc:482] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.218149 3587 raft_consensus.cc:504] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.218243 2263 catalog_manager.cc:5484] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3b61927599645d4b11278833b2f27d6 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "e3b61927599645d4b11278833b2f27d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.218701 3587 leader_election.cc:288] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:19.219094 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4acef14f154d0b8fb94428a45a380b" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:19.219180 2802 raft_consensus.cc:2454] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:19.219193 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4acef14f154d0b8fb94428a45a380b" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:19.219286 3063 raft_consensus.cc:2454] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:19.219504 2606 leader_election.cc:302] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 81b9495629bb458892b031db80bc3715, 886e4eb306064b98ab923623b8f1d401; no voters:
I20240503 07:59:19.220500 2540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4acef14f154d0b8fb94428a45a380b" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3" is_pre_election: true
I20240503 07:59:19.220932 2540 raft_consensus.cc:2454] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:19.221109 3587 raft_consensus.cc:2781] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.221164 3587 raft_consensus.cc:482] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.221200 3587 raft_consensus.cc:3037] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.221805 3587 raft_consensus.cc:504] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.222194 3587 leader_election.cc:288] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 election: Requested vote from peers e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:19.222405 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4acef14f154d0b8fb94428a45a380b" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:19.222494 3063 raft_consensus.cc:3037] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.222705 2802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4acef14f154d0b8fb94428a45a380b" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:19.222790 2802 raft_consensus.cc:3037] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.223224 3063 raft_consensus.cc:2454] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:19.223557 2802 raft_consensus.cc:2454] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:19.223819 2607 leader_election.cc:302] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 81b9495629bb458892b031db80bc3715, 886e4eb306064b98ab923623b8f1d401; no voters:
I20240503 07:59:19.223912 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4acef14f154d0b8fb94428a45a380b" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:19.223945 2540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4acef14f154d0b8fb94428a45a380b" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3"
I20240503 07:59:19.224025 2540 raft_consensus.cc:3037] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.224016 2933 raft_consensus.cc:2454] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 0.
I20240503 07:59:19.224165 2932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4acef14f154d0b8fb94428a45a380b" candidate_uuid: "886e4eb306064b98ab923623b8f1d401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:19.224227 2932 raft_consensus.cc:3037] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.224627 2540 raft_consensus.cc:2454] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:19.224779 3587 raft_consensus.cc:2781] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.224854 3587 raft_consensus.cc:686] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Becoming Leader. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Running, Role: LEADER
I20240503 07:59:19.224978 2932 raft_consensus.cc:2454] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886e4eb306064b98ab923623b8f1d401 in term 1.
I20240503 07:59:19.225193 3587 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } }
I20240503 07:59:19.226779 2263 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 886e4eb306064b98ab923623b8f1d401 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:19.355324 3625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:19.355509 3625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:19.358372 3625 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 07:59:19.446738 3063 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.447012 2932 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.447211 2802 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.447338 2540 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.447698 3532 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.447860 3569 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.447983 3390 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.448228 3532 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.453420 2670 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.453951 3654 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.454245 2801 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.460548 2932 raft_consensus.cc:1261] T 401e87d2d48e4515b725c10757178536 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.460893 3353 consensus_queue.cc:1035] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.461666 2539 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.462718 2409 raft_consensus.cc:1261] T 401e87d2d48e4515b725c10757178536 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.463155 3438 consensus_queue.cc:1035] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.466001 3063 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.466369 3580 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.466635 3580 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.467235 3654 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.467958 2669 raft_consensus.cc:1261] T 401e87d2d48e4515b725c10757178536 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.470178 3438 consensus_queue.cc:1035] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.474833 2800 raft_consensus.cc:1261] T 401e87d2d48e4515b725c10757178536 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.475191 3438 consensus_queue.cc:1035] T 401e87d2d48e4515b725c10757178536 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 165.634 ms
time per row: 0.082817 ms
I20240503 07:59:19.566025 2262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45856:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:19.571411 2518 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8af8c7edf45368869b33dc1819e35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.571416 2389 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8af8c7edf45368869b33dc1819e35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.571692 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8af8c7edf45368869b33dc1819e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.571707 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8af8c7edf45368869b33dc1819e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.571702 2781 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8af8c7edf45368869b33dc1819e35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.571938 2781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8af8c7edf45368869b33dc1819e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.573007 2913 tablet_service.cc:1467] Processing CreateTablet for tablet d116d60ce3a04fc782762c4b8808a178 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.573273 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116d60ce3a04fc782762c4b8808a178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.573330 2911 tablet_service.cc:1467] Processing CreateTablet for tablet dc9685543090435199e2c63872396b0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.573555 2911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9685543090435199e2c63872396b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.573864 3593 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:19.573873 3596 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:19.574689 3593 tablet_bootstrap.cc:654] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.574702 3596 tablet_bootstrap.cc:654] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.575295 3595 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:19.575778 3596 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:19.575853 3596 ts_tablet_manager.cc:1367] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.576014 3595 tablet_bootstrap.cc:654] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.576243 3596 raft_consensus.cc:348] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.576346 3596 raft_consensus.cc:374] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.576383 3596 raft_consensus.cc:729] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.576489 3596 consensus_queue.cc:260] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.576575 3594 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:19.576750 3596 ts_tablet_manager.cc:1398] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.577131 3042 tablet_service.cc:1467] Processing CreateTablet for tablet dc9685543090435199e2c63872396b0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.577093 3044 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8af8c7edf45368869b33dc1819e35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.577373 3042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9685543090435199e2c63872396b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.577378 3594 tablet_bootstrap.cc:654] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.577706 3044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8af8c7edf45368869b33dc1819e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.578709 3593 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:19.578784 3593 ts_tablet_manager.cc:1367] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:19.579128 3593 raft_consensus.cc:348] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.579214 3593 raft_consensus.cc:374] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.579241 3593 raft_consensus.cc:729] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.579331 3598 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:19.579339 3593 consensus_queue.cc:260] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.579313 3043 tablet_service.cc:1467] Processing CreateTablet for tablet d116d60ce3a04fc782762c4b8808a178 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.579572 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116d60ce3a04fc782762c4b8808a178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.579593 3593 ts_tablet_manager.cc:1398] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.580090 3598 tablet_bootstrap.cc:654] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.580813 2781 tablet_service.cc:1467] Processing CreateTablet for tablet d116d60ce3a04fc782762c4b8808a178 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.580878 3598 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:19.580946 3598 ts_tablet_manager.cc:1367] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:19.581048 2781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116d60ce3a04fc782762c4b8808a178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.581303 3598 raft_consensus.cc:348] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.581389 3598 raft_consensus.cc:374] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.581418 3598 raft_consensus.cc:729] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.581523 3598 consensus_queue.cc:260] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.581776 3598 ts_tablet_manager.cc:1398] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.581924 3598 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:19.581874 2649 tablet_service.cc:1467] Processing CreateTablet for tablet dc9685543090435199e2c63872396b0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.582057 2650 tablet_service.cc:1467] Processing CreateTablet for tablet d116d60ce3a04fc782762c4b8808a178 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.582120 2649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9685543090435199e2c63872396b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.582231 2650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116d60ce3a04fc782762c4b8808a178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.582614 3598 tablet_bootstrap.cc:654] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.583492 3598 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:19.583564 3598 ts_tablet_manager.cc:1367] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:19.584002 3601 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:19.584439 2387 tablet_service.cc:1467] Processing CreateTablet for tablet d116d60ce3a04fc782762c4b8808a178 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.584659 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116d60ce3a04fc782762c4b8808a178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.584751 3601 tablet_bootstrap.cc:654] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.581874 2651 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8af8c7edf45368869b33dc1819e35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.585351 2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8af8c7edf45368869b33dc1819e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.585531 3598 raft_consensus.cc:348] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.585626 3598 raft_consensus.cc:374] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.585654 3598 raft_consensus.cc:729] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.585793 3598 consensus_queue.cc:260] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.585934 2389 tablet_service.cc:1467] Processing CreateTablet for tablet dc9685543090435199e2c63872396b0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.586091 3598 ts_tablet_manager.cc:1398] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:19.586144 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9685543090435199e2c63872396b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.586591 3596 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:19.587225 3596 tablet_bootstrap.cc:654] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.587842 2518 tablet_service.cc:1467] Processing CreateTablet for tablet dc9685543090435199e2c63872396b0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.588052 2518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9685543090435199e2c63872396b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.588052 3596 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:19.588124 3596 ts_tablet_manager.cc:1367] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.588459 3596 raft_consensus.cc:348] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.588541 3596 raft_consensus.cc:374] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.588569 3596 raft_consensus.cc:729] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.588688 3596 consensus_queue.cc:260] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.588901 3596 ts_tablet_manager.cc:1398] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.589033 3596 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:19.589725 3593 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3: Bootstrap starting.
I20240503 07:59:19.589795 3596 tablet_bootstrap.cc:654] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.589825 3595 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:19.589898 3595 ts_tablet_manager.cc:1367] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 07:59:19.590299 3595 raft_consensus.cc:348] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.590409 3595 raft_consensus.cc:374] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.590441 3595 raft_consensus.cc:729] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.590521 3593 tablet_bootstrap.cc:654] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.590566 3595 consensus_queue.cc:260] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.590837 3595 ts_tablet_manager.cc:1398] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.590886 3596 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a: No bootstrap required, opened a new log
I20240503 07:59:19.590956 3596 ts_tablet_manager.cc:1367] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:19.590986 3595 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6: Bootstrap starting.
I20240503 07:59:19.591342 3596 raft_consensus.cc:348] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.591445 3596 raft_consensus.cc:374] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.591480 3596 raft_consensus.cc:729] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.591588 3596 consensus_queue.cc:260] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.591805 3595 tablet_bootstrap.cc:654] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.591853 3596 ts_tablet_manager.cc:1398] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:19.592242 3593 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3: No bootstrap required, opened a new log
I20240503 07:59:19.592314 3593 ts_tablet_manager.cc:1367] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:19.592635 3593 raft_consensus.cc:348] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.592746 3593 raft_consensus.cc:374] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.592777 3593 raft_consensus.cc:729] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.592801 3595 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6: No bootstrap required, opened a new log
I20240503 07:59:19.592875 3595 ts_tablet_manager.cc:1367] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.592882 3593 consensus_queue.cc:260] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.593166 3593 ts_tablet_manager.cc:1398] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.593315 3595 raft_consensus.cc:348] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.593416 3595 raft_consensus.cc:374] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.593448 3595 raft_consensus.cc:729] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b61927599645d4b11278833b2f27d6, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.593590 3595 consensus_queue.cc:260] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.593901 3595 ts_tablet_manager.cc:1398] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.595278 3598 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715: Bootstrap starting.
I20240503 07:59:19.595402 3601 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:19.595468 3601 ts_tablet_manager.cc:1367] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:19.595860 3601 raft_consensus.cc:348] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.595968 3601 raft_consensus.cc:374] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.595999 3601 raft_consensus.cc:729] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.596101 3598 tablet_bootstrap.cc:654] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.596108 3601 consensus_queue.cc:260] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.596380 3601 ts_tablet_manager.cc:1398] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.596551 3601 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:19.597378 3601 tablet_bootstrap.cc:654] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.598416 3601 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:19.598486 3601 ts_tablet_manager.cc:1367] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.598867 3601 raft_consensus.cc:348] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.598948 3601 raft_consensus.cc:374] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.598974 3601 raft_consensus.cc:729] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.599081 3601 consensus_queue.cc:260] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.599277 3601 ts_tablet_manager.cc:1398] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.599422 3601 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:19.600060 3601 tablet_bootstrap.cc:654] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.600934 3601 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: No bootstrap required, opened a new log
I20240503 07:59:19.601006 3601 ts_tablet_manager.cc:1367] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.601377 3594 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:19.601346 3601 raft_consensus.cc:348] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.601423 3601 raft_consensus.cc:374] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.601440 3601 raft_consensus.cc:729] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.601471 3594 ts_tablet_manager.cc:1367] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.000s
I20240503 07:59:19.601567 3601 consensus_queue.cc:260] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.601780 3601 ts_tablet_manager.cc:1398] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.601881 3594 raft_consensus.cc:348] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.601982 3594 raft_consensus.cc:374] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.602017 3594 raft_consensus.cc:729] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.602133 3594 consensus_queue.cc:260] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.602362 3594 ts_tablet_manager.cc:1398] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.604173 3594 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21: Bootstrap starting.
I20240503 07:59:19.605042 3594 tablet_bootstrap.cc:654] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.606035 3594 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21: No bootstrap required, opened a new log
I20240503 07:59:19.606107 3594 ts_tablet_manager.cc:1367] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:19.606622 3594 raft_consensus.cc:348] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.606710 3594 raft_consensus.cc:374] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.606739 3594 raft_consensus.cc:729] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.606844 3594 consensus_queue.cc:260] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.607100 3594 ts_tablet_manager.cc:1398] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:19.617338 3598 tablet_bootstrap.cc:492] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715: No bootstrap required, opened a new log
I20240503 07:59:19.617435 3598 ts_tablet_manager.cc:1367] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20240503 07:59:19.617815 3598 raft_consensus.cc:348] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.617903 3598 raft_consensus.cc:374] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.617925 3598 raft_consensus.cc:729] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b9495629bb458892b031db80bc3715, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.618034 3598 consensus_queue.cc:260] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.618264 3598 ts_tablet_manager.cc:1398] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:19.647006 2998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:38745: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 07:59:19.683768 3608 raft_consensus.cc:482] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.683889 3608 raft_consensus.cc:504] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.684540 3608 leader_election.cc:288] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:19.684772 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8af8c7edf45368869b33dc1819e35" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:19.684880 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8af8c7edf45368869b33dc1819e35" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3" is_pre_election: true
I20240503 07:59:19.684916 2409 raft_consensus.cc:2454] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 0.
I20240503 07:59:19.685027 2539 raft_consensus.cc:2454] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 0.
I20240503 07:59:19.685330 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8af8c7edf45368869b33dc1819e35" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:19.685458 2670 raft_consensus.cc:2454] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 0.
I20240503 07:59:19.685775 2735 leader_election.cc:302] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 3631cd1b9b79487581e9d31178f671c3, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:19.685951 3608 raft_consensus.cc:2781] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.686017 3608 raft_consensus.cc:482] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.686069 3608 raft_consensus.cc:3037] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.687171 3608 raft_consensus.cc:504] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.687697 3608 leader_election.cc:288] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [CANDIDATE]: Term 1 election: Requested vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:19.687832 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8af8c7edf45368869b33dc1819e35" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:19.687948 2409 raft_consensus.cc:3037] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.687973 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8af8c7edf45368869b33dc1819e35" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3631cd1b9b79487581e9d31178f671c3"
I20240503 07:59:19.688081 2539 raft_consensus.cc:3037] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.688192 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8af8c7edf45368869b33dc1819e35" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:19.688277 2670 raft_consensus.cc:3037] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.688855 2409 raft_consensus.cc:2454] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 1.
I20240503 07:59:19.689198 2539 raft_consensus.cc:2454] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 1.
I20240503 07:59:19.689390 2670 raft_consensus.cc:2454] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 1.
I20240503 07:59:19.689577 2735 leader_election.cc:302] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, 3631cd1b9b79487581e9d31178f671c3, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:19.689599 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8af8c7edf45368869b33dc1819e35" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:19.689677 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8af8c7edf45368869b33dc1819e35" candidate_uuid: "0bc088d2657a49b39cb7981f0e3eec21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:19.689738 3062 raft_consensus.cc:3037] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.689755 3608 raft_consensus.cc:2781] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.689913 3608 raft_consensus.cc:686] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc088d2657a49b39cb7981f0e3eec21, State: Running, Role: LEADER
I20240503 07:59:19.690070 3608 consensus_queue.cc:237] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.690701 3062 raft_consensus.cc:2454] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc088d2657a49b39cb7981f0e3eec21 in term 1.
I20240503 07:59:19.691812 2259 catalog_manager.cc:5484] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.716537 3392 raft_consensus.cc:482] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.716632 3392 raft_consensus.cc:504] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.717767 3392 leader_election.cc:288] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:19.722656 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116d60ce3a04fc782762c4b8808a178" candidate_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:19.722796 3062 raft_consensus.cc:2454] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd82dbc60d0f40688fbe8e712dc7107a in term 0.
I20240503 07:59:19.723112 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116d60ce3a04fc782762c4b8808a178" candidate_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:19.723237 2670 raft_consensus.cc:2454] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd82dbc60d0f40688fbe8e712dc7107a in term 0.
I20240503 07:59:19.723600 2344 leader_election.cc:302] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 81b9495629bb458892b031db80bc3715, 886e4eb306064b98ab923623b8f1d401, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:19.723856 3392 raft_consensus.cc:2781] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.723924 3392 raft_consensus.cc:482] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.723960 3392 raft_consensus.cc:3037] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.724330 2800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116d60ce3a04fc782762c4b8808a178" candidate_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21" is_pre_election: true
I20240503 07:59:19.724453 2800 raft_consensus.cc:2454] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd82dbc60d0f40688fbe8e712dc7107a in term 0.
I20240503 07:59:19.725524 2932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116d60ce3a04fc782762c4b8808a178" candidate_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:19.725644 2932 raft_consensus.cc:2454] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd82dbc60d0f40688fbe8e712dc7107a in term 0.
I20240503 07:59:19.741804 3392 raft_consensus.cc:504] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.742266 3392 leader_election.cc:288] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [CANDIDATE]: Term 1 election: Requested vote from peers 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:19.742540 2800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116d60ce3a04fc782762c4b8808a178" candidate_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc088d2657a49b39cb7981f0e3eec21"
I20240503 07:59:19.742591 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116d60ce3a04fc782762c4b8808a178" candidate_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:19.742579 2932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116d60ce3a04fc782762c4b8808a178" candidate_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:19.742662 2800 raft_consensus.cc:3037] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.742691 2932 raft_consensus.cc:3037] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.742686 3062 raft_consensus.cc:3037] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.743111 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116d60ce3a04fc782762c4b8808a178" candidate_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:19.743221 2670 raft_consensus.cc:3037] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.743568 2800 raft_consensus.cc:2454] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd82dbc60d0f40688fbe8e712dc7107a in term 1.
I20240503 07:59:19.743568 3062 raft_consensus.cc:2454] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd82dbc60d0f40688fbe8e712dc7107a in term 1.
I20240503 07:59:19.743580 2932 raft_consensus.cc:2454] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd82dbc60d0f40688fbe8e712dc7107a in term 1.
I20240503 07:59:19.743906 2345 leader_election.cc:302] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0bc088d2657a49b39cb7981f0e3eec21, cd82dbc60d0f40688fbe8e712dc7107a, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:19.744081 3392 raft_consensus.cc:2781] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.744108 2670 raft_consensus.cc:2454] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd82dbc60d0f40688fbe8e712dc7107a in term 1.
I20240503 07:59:19.744279 3392 raft_consensus.cc:686] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEADER]: Becoming Leader. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Running, Role: LEADER
I20240503 07:59:19.744590 3392 consensus_queue.cc:237] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } }
I20240503 07:59:19.747026 2259 catalog_manager.cc:5484] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: term changed from 0 to 1, leader changed from <none> to cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.795183 3351 raft_consensus.cc:482] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.795333 3351 raft_consensus.cc:504] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.795948 3351 leader_election.cc:288] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:19.796263 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9685543090435199e2c63872396b0f" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401" is_pre_election: true
I20240503 07:59:19.796265 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9685543090435199e2c63872396b0f" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" is_pre_election: true
I20240503 07:59:19.796360 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9685543090435199e2c63872396b0f" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715" is_pre_election: true
I20240503 07:59:19.796404 2409 raft_consensus.cc:2454] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:19.796406 2670 raft_consensus.cc:2454] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:19.796442 2932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9685543090435199e2c63872396b0f" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6" is_pre_election: true
I20240503 07:59:19.796494 3062 raft_consensus.cc:2454] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:19.796553 2932 raft_consensus.cc:2454] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 0.
I20240503 07:59:19.796702 2473 leader_election.cc:302] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, 886e4eb306064b98ab923623b8f1d401, cd82dbc60d0f40688fbe8e712dc7107a; no voters:
I20240503 07:59:19.796885 3351 raft_consensus.cc:2781] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.796943 3351 raft_consensus.cc:482] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.796968 3351 raft_consensus.cc:3037] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.797796 3351 raft_consensus.cc:504] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.798167 3351 leader_election.cc:288] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Requested vote from peers cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251), 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343), 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145), e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:19.798457 2932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9685543090435199e2c63872396b0f" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3b61927599645d4b11278833b2f27d6"
I20240503 07:59:19.798460 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9685543090435199e2c63872396b0f" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "886e4eb306064b98ab923623b8f1d401"
I20240503 07:59:19.798480 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9685543090435199e2c63872396b0f" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd82dbc60d0f40688fbe8e712dc7107a"
I20240503 07:59:19.798550 2932 raft_consensus.cc:3037] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.798560 2670 raft_consensus.cc:3037] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.798540 3062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9685543090435199e2c63872396b0f" candidate_uuid: "3631cd1b9b79487581e9d31178f671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b9495629bb458892b031db80bc3715"
I20240503 07:59:19.798565 2409 raft_consensus.cc:3037] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.798626 3062 raft_consensus.cc:3037] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.799397 2932 raft_consensus.cc:2454] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:19.799397 2409 raft_consensus.cc:2454] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:19.799470 3062 raft_consensus.cc:2454] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:19.799470 2670 raft_consensus.cc:2454] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3631cd1b9b79487581e9d31178f671c3 in term 1.
I20240503 07:59:19.799722 2473 leader_election.cc:302] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3631cd1b9b79487581e9d31178f671c3, cd82dbc60d0f40688fbe8e712dc7107a, e3b61927599645d4b11278833b2f27d6; no voters:
I20240503 07:59:19.799870 3351 raft_consensus.cc:2781] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.799988 3351 raft_consensus.cc:686] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3631cd1b9b79487581e9d31178f671c3, State: Running, Role: LEADER
I20240503 07:59:19.800155 3351 consensus_queue.cc:237] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } }
I20240503 07:59:19.801858 2259 catalog_manager.cc:5484] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:19.928181 3699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:19.928387 3699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:19.931356 3699 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 07:59:20.005287 2409 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.005411 2670 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.005538 3062 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.005764 2539 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.005980 2800 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.006116 3608 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.006659 3689 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.006815 3608 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.007010 2670 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.007346 3689 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.007230 3730 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.007786 2932 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.013245 3064 raft_consensus.cc:1261] T dc9685543090435199e2c63872396b0f P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.014606 3351 consensus_queue.cc:1035] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.014986 2409 raft_consensus.cc:1261] T dc9685543090435199e2c63872396b0f P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.015626 3730 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.017045 3062 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.017112 3503 consensus_queue.cc:1035] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.017910 3729 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.018148 2671 raft_consensus.cc:1261] T dc9685543090435199e2c63872396b0f P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.018208 3729 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.019361 3503 consensus_queue.cc:1035] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.023062 2932 raft_consensus.cc:1261] T dc9685543090435199e2c63872396b0f P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.023835 3503 consensus_queue.cc:1035] T dc9685543090435199e2c63872396b0f P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.033227 3384 mvcc.cc:205] Tried to move back new op lower bound from 7023506063407349760 to 7023506062543159296. Current Snapshot: MvccSnapshot[applied={T|T < 7023506063407349760}]
I20240503 07:59:20.051823 3584 mvcc.cc:205] Tried to move back new op lower bound from 7023506063447453696 to 7023506062543159296. Current Snapshot: MvccSnapshot[applied={T|T < 7023506063407349760}]
I20240503 07:59:20.055501 3419 mvcc.cc:205] Tried to move back new op lower bound from 7023506063375998976 to 7023506062316642304. Current Snapshot: MvccSnapshot[applied={T|T < 7023506063375998976 or (T in {7023506063375998976})}]
I20240503 07:59:20.070937 3748 mvcc.cc:205] Tried to move back new op lower bound from 7023506063375998976 to 7023506062316642304. Current Snapshot: MvccSnapshot[applied={T|T < 7023506063375998976}]
I20240503 07:59:20.092300 3403 mvcc.cc:205] Tried to move back new op lower bound from 7023506063375998976 to 7023506062316642304. Current Snapshot: MvccSnapshot[applied={T|T < 7023506063375998976}]
INSERT report
rows total: 2000
time total: 155.823 ms
time per row: 0.0779115 ms
I20240503 07:59:20.127627 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:38745
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:42701
--webserver_port=40585
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:20.231916 3763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:20.232103 3763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:20.232143 3763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:20.235150 3763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:20.235208 3763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:20.235286 3763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:20.238689 3763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:38745
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=40585
--tserver_master_addrs=127.0.99.190:42701
--never_fsync=true
--heap_profile_path=/tmp/kudu.3763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:20.239444 3763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:20.240480 3763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:20.248199 3771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:20.248817 3770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:20.249331 3773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:20.250329 3763 server_base.cc:1034] running on GCE node
I20240503 07:59:20.250643 3763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:20.251158 3763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:20.252286 3763 hybrid_clock.cc:648] HybridClock initialized: now 1714723160252271 us; error 32 us; skew 500 ppm
I20240503 07:59:20.257251 3763 webserver.cc:460] Webserver started at http://127.0.99.135:40585/ using document root <none> and password file <none>
I20240503 07:59:20.257737 3763 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:20.257812 3763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:20.260885 3763 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 07:59:20.262624 3779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:20.263511 3763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:20.263602 3763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6338e74c39eb4583842e387e64fce522"
format_stamp: "Formatted at 2024-05-03 07:59:13 on dist-test-slave-7b05"
I20240503 07:59:20.263979 3763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:20.280555 3763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:20.281123 3763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:20.281255 3763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:20.281803 3763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:20.282758 3786 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 07:59:20.285703 3763 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 07:59:20.285777 3763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 07:59:20.285830 3763 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 07:59:20.287108 3763 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 07:59:20.287159 3763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.287384 3786 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:20.297734 3763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:38745
I20240503 07:59:20.297744 3893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:38745 every 8 connection(s)
I20240503 07:59:20.299026 3763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:20.302503 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 3763
I20240503 07:59:20.302657 398 test_util.cc:268] Using random seed: 417047617
I20240503 07:59:20.303071 3786 log.cc:824] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:20.304798 3786 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:20.305116 3786 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:20.306243 3786 ts_tablet_manager.cc:1367] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20240503 07:59:20.308765 3894 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42701
I20240503 07:59:20.308977 3894 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:20.309554 3894 heartbeater.cc:511] Master 127.0.99.190:42701 requested a full tablet report, sending...
I20240503 07:59:20.309325 3786 raft_consensus.cc:348] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:20.309684 3786 raft_consensus.cc:729] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.310138 3786 consensus_queue.cc:260] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } }
I20240503 07:59:20.311836 2259 ts_manager.cc:194] Re-registered known tserver with Master: 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745)
I20240503 07:59:20.311848 3786 ts_tablet_manager.cc:1398] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:59:20.312734 2259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:49161
I20240503 07:59:20.314468 3894 heartbeater.cc:503] Master 127.0.99.190:42701 was elected leader, sending a full tablet report...
I20240503 07:59:20.315624 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 07:59:20.315816 398 test_util.cc:268] Using random seed: 417060775
I20240503 07:59:20.323902 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 07:59:20.324040 398 test_util.cc:268] Using random seed: 417068999
I20240503 07:59:20.338213 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 07:59:20.338352 398 test_util.cc:268] Using random seed: 417083311
I20240503 07:59:20.349828 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 07:59:20.349961 398 test_util.cc:268] Using random seed: 417094920
I20240503 07:59:20.359998 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240503 07:59:20.549899 3895 tablet.cc:2369] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:21.263260 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:21.357620 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:21.358808 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:21.362380 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:21.369377 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:21.374128 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:21.404860 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 07:59:22.074842 2473 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.129:38251, user_credentials={real_user=slave}} blocked reactor thread for 46290.1us
I20240503 07:59:22.075347 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:22.103705 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:22.106627 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:22.119521 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:22.125279 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:22.160730 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:22.183538 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:22.431433 3058 consensus_queue.cc:237] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:22.433673 2932 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240503 07:59:22.434289 2801 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240503 07:59:22.434372 2408 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240503 07:59:22.434728 3933 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
W20240503 07:59:22.435750 2998 consensus_peers.cc:485] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df62f8897f4f4e4abe13f3455a665031. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:22.435904 3932 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20240503 07:59:22.436707 4028 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 46, Time since last communication: 0.000s
I20240503 07:59:22.437215 2668 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240503 07:59:22.440419 3932 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 45, Time since last communication: 0.000s
I20240503 07:59:22.440307 4028 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.442530 2408 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.449824 2799 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.450071 2261 catalog_manager.cc:5484] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index -1 to 48, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:22.453891 2929 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.471151 2668 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.572582 2670 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 1.46, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:22.575311 2930 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 1 index: 47. (index mismatch)
I20240503 07:59:22.575745 3987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20240503 07:59:22.578141 2802 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 1 index: 47. (index mismatch)
I20240503 07:59:22.581707 3971 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20240503 07:59:22.583456 3058 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 1 index: 47. (index mismatch)
I20240503 07:59:22.584741 3532 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 45, Time since last communication: 0.000s
I20240503 07:59:22.589012 2536 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 1 index: 47. (index mismatch)
I20240503 07:59:22.590466 3971 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20240503 07:59:22.598258 3532 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.601187 2929 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.614725 2799 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.615377 3060 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.658603 2800 consensus_queue.cc:237] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:22.661903 2405 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240503 07:59:22.662317 3993 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20240503 07:59:22.662369 2537 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240503 07:59:22.662732 3993 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20240503 07:59:22.664386 3993 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.666121 3064 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240503 07:59:22.666440 2671 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240503 07:59:22.666620 4080 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 39, Time since last communication: 0.000s
I20240503 07:59:22.668426 2405 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.668706 2536 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.672569 3064 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.672732 2258 catalog_manager.cc:5484] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index -1 to 41, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.674142 3060 raft_consensus.cc:1203] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40] Dedup: 1.40->[]
I20240503 07:59:22.675370 3689 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 39, Time since last communication: 0.000s
I20240503 07:59:22.679467 2670 raft_consensus.cc:1203] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.42] Dedup: 1.40->[1.41-1.42]
I20240503 07:59:22.683382 2670 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.686036 2800 consensus_queue.cc:237] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:22.688176 2930 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240503 07:59:22.688184 2405 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240503 07:59:22.688475 2536 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240503 07:59:22.688530 3689 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240503 07:59:22.688840 3993 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240503 07:59:22.688984 3755 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240503 07:59:22.690752 2667 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240503 07:59:22.691162 3689 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.692061 2930 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.693380 4080 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240503 07:59:22.693275 2536 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.694823 2405 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.695806 2671 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
W20240503 07:59:22.697293 2736 consensus_peers.cc:485] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a11003820e14dc7b2615f20baf179cc. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:22.697446 2736 consensus_peers.cc:485] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebe8af8c7edf45368869b33dc1819e35. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:22.712471 2536 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.743683 2537 consensus_queue.cc:237] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 1.44, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:22.746094 2406 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240503 07:59:22.746191 2929 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240503 07:59:22.746695 4085 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20240503 07:59:22.747179 4085 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20240503 07:59:22.750092 3503 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.751902 2406 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.755693 2930 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
W20240503 07:59:22.782876 2605 consensus_peers.cc:485] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a4acef14f154d0b8fb94428a45a380b. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:22.786859 2474 consensus_peers.cc:485] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d732123aab240c4b4216f451e87de09. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:22.792063 2259 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 reported cstate change: config changed from index -1 to 47, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.792332 2259 catalog_manager.cc:5484] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 reported cstate change: config changed from index -1 to 45, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.792588 2259 catalog_manager.cc:5484] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 reported cstate change: config changed from index -1 to 37, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.797880 2667 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240503 07:59:22.801906 3064 consensus_queue.cc:237] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: true } }
I20240503 07:59:22.803741 2409 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240503 07:59:22.803759 2802 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240503 07:59:22.804142 3933 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 07:59:22.804282 3571 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 07:59:22.805876 4085 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 43, Time since last communication: 0.000s
I20240503 07:59:22.807315 3933 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: true } } }
I20240503 07:59:22.809381 2802 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: true } } }
W20240503 07:59:22.810770 2999 consensus_peers.cc:485] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 -> Peer 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343): Couldn't send request to peer 886e4eb306064b98ab923623b8f1d401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9d779a081cd48628e24d1321f7b42af. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:22.810700 2258 catalog_manager.cc:5484] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index -1 to 50, NON_VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:22.811338 2409 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: true } } }
I20240503 07:59:22.812809 2670 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.813941 2540 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240503 07:59:22.818070 3934 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 07:59:22.818867 2540 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: true } } }
I20240503 07:59:22.838832 3064 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 42. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240503 07:59:22.845039 2931 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240503 07:59:22.848095 3932 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 47, Time since last communication: 0.000s
I20240503 07:59:22.859498 2931 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: true } } }
I20240503 07:59:22.887773 4002 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 42, Time since last communication: 0.000s
I20240503 07:59:22.923898 3063 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:22.959255 4199 ts_tablet_manager.cc:902] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:22.965737 4199 tablet_copy_client.cc:308] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:36145
I20240503 07:59:22.988612 3084 tablet_copy_service.cc:140] P 81b9495629bb458892b031db80bc3715: Received BeginTabletCopySession request for tablet df62f8897f4f4e4abe13f3455a665031 from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:33237)
I20240503 07:59:22.988773 3084 tablet_copy_service.cc:161] P 81b9495629bb458892b031db80bc3715: Beginning new tablet copy session on tablet df62f8897f4f4e4abe13f3455a665031 from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:33237: session id = 6338e74c39eb4583842e387e64fce522-df62f8897f4f4e4abe13f3455a665031
I20240503 07:59:22.990341 3084 tablet_copy_source_session.cc:205] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:22.993130 4199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df62f8897f4f4e4abe13f3455a665031. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.003154 4199 tablet_copy_client.cc:783] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:23.003448 4199 tablet_copy_client.cc:647] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:23.005357 4199 tablet_copy_client.cc:523] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:23.009891 4199 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:23.188552 4199 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=60 overwritten=0 applied=60 ignored=0} inserts{seen=732 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:23.191169 4199 tablet_bootstrap.cc:492] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:23.191968 4199 ts_tablet_manager.cc:1367] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.182s user 0.064s sys 0.006s
I20240503 07:59:23.193730 4199 raft_consensus.cc:348] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:23.193982 4199 raft_consensus.cc:729] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:23.194128 4199 consensus_queue.cc:260] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:23.194516 4199 ts_tablet_manager.cc:1398] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.195693 3084 tablet_copy_service.cc:342] P 81b9495629bb458892b031db80bc3715: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-df62f8897f4f4e4abe13f3455a665031 received from {username='slave'} at 127.0.99.135:33237
I20240503 07:59:23.195780 3084 tablet_copy_service.cc:433] P 81b9495629bb458892b031db80bc3715: ending tablet copy session 6338e74c39eb4583842e387e64fce522-df62f8897f4f4e4abe13f3455a665031 on tablet df62f8897f4f4e4abe13f3455a665031 with peer 6338e74c39eb4583842e387e64fce522
I20240503 07:59:23.197342 4199 ts_tablet_manager.cc:902] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199)
I20240503 07:59:23.197778 4199 tablet_copy_client.cc:308] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37199
I20240503 07:59:23.203845 4215 ts_tablet_manager.cc:902] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:23.206418 4215 tablet_copy_client.cc:308] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44343
I20240503 07:59:23.214115 2820 tablet_copy_service.cc:140] P 0bc088d2657a49b39cb7981f0e3eec21: Received BeginTabletCopySession request for tablet ebe8af8c7edf45368869b33dc1819e35 from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:57215)
I20240503 07:59:23.214264 2820 tablet_copy_service.cc:161] P 0bc088d2657a49b39cb7981f0e3eec21: Beginning new tablet copy session on tablet ebe8af8c7edf45368869b33dc1819e35 from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:57215: session id = 6338e74c39eb4583842e387e64fce522-ebe8af8c7edf45368869b33dc1819e35
I20240503 07:59:23.215746 2820 tablet_copy_source_session.cc:205] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:23.217249 4199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8af8c7edf45368869b33dc1819e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.219348 4199 tablet_copy_client.cc:783] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:23.219502 4199 tablet_copy_client.cc:647] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:23.224978 2691 tablet_copy_service.cc:140] P 886e4eb306064b98ab923623b8f1d401: Received BeginTabletCopySession request for tablet 0a4acef14f154d0b8fb94428a45a380b from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:33563)
I20240503 07:59:23.225142 2691 tablet_copy_service.cc:161] P 886e4eb306064b98ab923623b8f1d401: Beginning new tablet copy session on tablet 0a4acef14f154d0b8fb94428a45a380b from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:33563: session id = 6338e74c39eb4583842e387e64fce522-0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:23.226050 4199 tablet_copy_client.cc:523] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:23.226508 2691 tablet_copy_source_session.cc:205] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:23.227777 4199 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:23.248695 4227 ts_tablet_manager.cc:902] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199)
I20240503 07:59:23.249250 4227 tablet_copy_client.cc:308] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37199
I20240503 07:59:23.249861 2820 tablet_copy_service.cc:140] P 0bc088d2657a49b39cb7981f0e3eec21: Received BeginTabletCopySession request for tablet 6a11003820e14dc7b2615f20baf179cc from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:57215)
I20240503 07:59:23.249967 2820 tablet_copy_service.cc:161] P 0bc088d2657a49b39cb7981f0e3eec21: Beginning new tablet copy session on tablet 6a11003820e14dc7b2615f20baf179cc from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:57215: session id = 6338e74c39eb4583842e387e64fce522-6a11003820e14dc7b2615f20baf179cc
I20240503 07:59:23.250864 2820 tablet_copy_source_session.cc:205] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:23.251879 4227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a11003820e14dc7b2615f20baf179cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.276571 4227 tablet_copy_client.cc:783] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:23.277408 4227 tablet_copy_client.cc:647] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:23.280886 3848 raft_consensus.cc:1203] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.47->[1.48-1.61] Dedup: 1.60->[1.61-1.61]
I20240503 07:59:23.290529 4227 tablet_copy_client.cc:523] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:23.291858 4227 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:23.364013 4199 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=52 ignored=0} inserts{seen=737 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:23.364360 4199 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:23.364579 4199 ts_tablet_manager.cc:1367] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.137s user 0.049s sys 0.012s
I20240503 07:59:23.365020 4199 raft_consensus.cc:348] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:23.365248 4199 raft_consensus.cc:729] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:23.365401 4199 consensus_queue.cc:260] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 52, Last appended: 1.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:23.366421 4199 ts_tablet_manager.cc:1398] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.367158 2820 tablet_copy_service.cc:342] P 0bc088d2657a49b39cb7981f0e3eec21: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-ebe8af8c7edf45368869b33dc1819e35 received from {username='slave'} at 127.0.99.135:57215
I20240503 07:59:23.367260 2820 tablet_copy_service.cc:433] P 0bc088d2657a49b39cb7981f0e3eec21: ending tablet copy session 6338e74c39eb4583842e387e64fce522-ebe8af8c7edf45368869b33dc1819e35 on tablet ebe8af8c7edf45368869b33dc1819e35 with peer 6338e74c39eb4583842e387e64fce522
I20240503 07:59:23.368750 4199 ts_tablet_manager.cc:902] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355)
I20240503 07:59:23.369207 4199 tablet_copy_client.cc:308] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40355
I20240503 07:59:23.404474 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4acef14f154d0b8fb94428a45a380b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.471974 2560 tablet_copy_service.cc:140] P 3631cd1b9b79487581e9d31178f671c3: Received BeginTabletCopySession request for tablet 7d732123aab240c4b4216f451e87de09 from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:58999)
I20240503 07:59:23.472143 2560 tablet_copy_service.cc:161] P 3631cd1b9b79487581e9d31178f671c3: Beginning new tablet copy session on tablet 7d732123aab240c4b4216f451e87de09 from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:58999: session id = 6338e74c39eb4583842e387e64fce522-7d732123aab240c4b4216f451e87de09
I20240503 07:59:23.473635 2560 tablet_copy_source_session.cc:205] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:23.476147 4199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d732123aab240c4b4216f451e87de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.479084 4199 tablet_copy_client.cc:783] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:23.479233 4199 tablet_copy_client.cc:647] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:23.481576 4199 tablet_copy_client.cc:523] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:23.483435 4199 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:23.484524 3847 raft_consensus.cc:1203] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.40->[1.41-1.57] Dedup: 1.52->[1.53-1.57]
I20240503 07:59:23.485198 4215 tablet_copy_client.cc:783] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:23.485330 4215 tablet_copy_client.cc:647] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:23.498328 4266 ts_tablet_manager.cc:902] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: Initiating tablet copy from peer 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:23.499150 4266 tablet_copy_client.cc:308] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:36145
I20240503 07:59:23.499658 3084 tablet_copy_service.cc:140] P 81b9495629bb458892b031db80bc3715: Received BeginTabletCopySession request for tablet c9d779a081cd48628e24d1321f7b42af from peer 886e4eb306064b98ab923623b8f1d401 ({username='slave'} at 127.0.99.131:60055)
I20240503 07:59:23.499751 3084 tablet_copy_service.cc:161] P 81b9495629bb458892b031db80bc3715: Beginning new tablet copy session on tablet c9d779a081cd48628e24d1321f7b42af from peer 886e4eb306064b98ab923623b8f1d401 at {username='slave'} at 127.0.99.131:60055: session id = 886e4eb306064b98ab923623b8f1d401-c9d779a081cd48628e24d1321f7b42af
I20240503 07:59:23.500692 3084 tablet_copy_source_session.cc:205] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:23.505016 4266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d779a081cd48628e24d1321f7b42af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.506561 4215 tablet_copy_client.cc:523] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:23.508118 4215 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:23.516415 4266 tablet_copy_client.cc:783] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:23.516640 4266 tablet_copy_client.cc:647] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:23.521800 4266 tablet_copy_client.cc:523] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:23.534746 4266 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: Bootstrap starting.
I20240503 07:59:23.548667 4184 raft_consensus.cc:1050] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:23.549186 4184 consensus_queue.cc:237] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:23.552626 2669 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 07:59:23.552650 3847 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 07:59:23.553131 4080 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 07:59:23.554822 4080 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 07:59:23.558030 2540 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 07:59:23.558862 3059 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 07:59:23.559242 3689 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 07:59:23.559360 4080 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 07:59:23.560746 3755 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index 41 to 59, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.562521 2540 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 41 to 59, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.564354 3064 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 41 to 59, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.565244 2263 catalog_manager.cc:5484] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index 41 to 59, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.570194 3847 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 41 to 59, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.577864 2406 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 07:59:23.578650 2669 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 41 to 59, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.584043 4080 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 07:59:23.617568 2406 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 41 to 59, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.636519 3934 raft_consensus.cc:1050] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:23.636976 3934 consensus_queue.cc:237] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:23.643875 2930 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 07:59:23.649709 3934 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 07:59:23.660351 3847 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 07:59:23.661266 3934 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 07:59:23.663487 2802 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 07:59:23.664446 3934 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 67, Time since last communication: 0.000s
I20240503 07:59:23.681015 2409 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 07:59:23.684692 3934 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 07:59:23.692312 4028 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 48 to 70, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.695425 2800 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 48 to 70, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.695520 2409 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 48 to 70, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.697019 2932 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 48 to 70, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.702207 2267 catalog_manager.cc:5484] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 reported cstate change: config changed from index 48 to 70, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.708957 4199 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=60 overwritten=0 applied=60 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:23.709297 4199 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:23.709534 4199 ts_tablet_manager.cc:1367] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.226s user 0.058s sys 0.003s
I20240503 07:59:23.709976 4215 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=54 ignored=0} inserts{seen=685 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:23.710089 4199 raft_consensus.cc:348] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:23.710275 4199 raft_consensus.cc:729] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:23.710426 4199 consensus_queue.cc:260] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:23.710765 4199 ts_tablet_manager.cc:1398] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:23.710237 4215 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:23.711094 4215 ts_tablet_manager.cc:1367] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.203s user 0.059s sys 0.000s
I20240503 07:59:23.711337 2560 tablet_copy_service.cc:342] P 3631cd1b9b79487581e9d31178f671c3: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-7d732123aab240c4b4216f451e87de09 received from {username='slave'} at 127.0.99.135:58999
I20240503 07:59:23.711438 2560 tablet_copy_service.cc:433] P 3631cd1b9b79487581e9d31178f671c3: ending tablet copy session 6338e74c39eb4583842e387e64fce522-7d732123aab240c4b4216f451e87de09 on tablet 7d732123aab240c4b4216f451e87de09 with peer 6338e74c39eb4583842e387e64fce522
I20240503 07:59:23.711350 3847 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 48 to 70, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.711475 4215 raft_consensus.cc:348] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:23.711575 4215 raft_consensus.cc:729] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:23.711656 4215 consensus_queue.cc:260] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 1.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:23.712092 4215 ts_tablet_manager.cc:1398] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.712574 2691 tablet_copy_service.cc:342] P 886e4eb306064b98ab923623b8f1d401: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-0a4acef14f154d0b8fb94428a45a380b received from {username='slave'} at 127.0.99.135:33563
I20240503 07:59:23.712669 2691 tablet_copy_service.cc:433] P 886e4eb306064b98ab923623b8f1d401: ending tablet copy session 6338e74c39eb4583842e387e64fce522-0a4acef14f154d0b8fb94428a45a380b on tablet 0a4acef14f154d0b8fb94428a45a380b with peer 6338e74c39eb4583842e387e64fce522
I20240503 07:59:23.726171 3059 consensus_queue.cc:237] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:23.727558 2800 consensus_queue.cc:237] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:23.727989 2799 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 07:59:23.728348 3934 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240503 07:59:23.730154 2931 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 07:59:23.730507 3846 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 07:59:23.730536 3934 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240503 07:59:23.731592 3059 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:23.731709 3934 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240503 07:59:23.732192 4184 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:23.732466 3848 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:23.733402 2669 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:23.733987 4184 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:23.735276 4184 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:23.735857 3571 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index 70 to 71, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.736392 4176 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index 59 to 65, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.738464 2253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df62f8897f4f4e4abe13f3455a665031 with cas_config_opid_index 70: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:23.738970 2254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ebe8af8c7edf45368869b33dc1819e35 with cas_config_opid_index 59: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:23.739898 2265 catalog_manager.cc:5484] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: config changed from index 59 to 65, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:23.740430 2931 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 70 to 71, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.742444 2259 catalog_manager.cc:5484] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 70 to 71, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:23.744499 2389 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe8af8c7edf45368869b33dc1819e35 with delete_type TABLET_DATA_TOMBSTONED (TS cd82dbc60d0f40688fbe8e712dc7107a not found in new config with opid_index 65) from {username='slave'} at 127.0.0.1:59282
I20240503 07:59:23.745780 2406 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 07:59:23.746511 2671 raft_consensus.cc:1261] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 07:59:23.746505 3846 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 70 to 71, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.747743 2650 tablet_service.cc:1514] Processing DeleteTablet for tablet df62f8897f4f4e4abe13f3455a665031 with delete_type TABLET_DATA_TOMBSTONED (TS 886e4eb306064b98ab923623b8f1d401 not found in new config with opid_index 71) from {username='slave'} at 127.0.0.1:52688
I20240503 07:59:23.747789 3934 consensus_queue.cc:1035] T df62f8897f4f4e4abe13f3455a665031 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240503 07:59:23.748839 2408 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 70 to 71, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.754530 3846 raft_consensus.cc:1203] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.46->[1.47-1.62] Dedup: 1.54->[1.55-1.62]
I20240503 07:59:23.754581 4286 tablet_replica.cc:330] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: stopping tablet replica
I20240503 07:59:23.754853 4286 raft_consensus.cc:2229] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.754979 4286 pending_rounds.cc:62] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: Trying to abort 1 pending ops.
I20240503 07:59:23.755030 4286 pending_rounds.cc:68] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: Aborting op as it isn't in flight: id { term: 1 index: 64 } timestamp: 7023506078599397376 op_type: WRITE_OP write_request { tablet_id: "ebe8af8c7edf45368869b33dc1819e35" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\254W\2261\021J\263\316\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""dd105aa0395b9a90" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506078575960064 authz_token { token_data: "\010\204\265\322\261\006\"3\n\005slave\022*\n a67e4da0a0be4cb2b0ca1abeb334fd39\020\001\030\001 \001(\001" signature: "<redacted>""\200\264\270>%GbuR6\313\312:u\344\333\207\345\200|&\326\314\275\360\341\n\210\337\247\034D;\304M\362\tU\226\031\006I\257\344\217>\267\023\363\013l\272,\340\006\326A\336\2730-\345\037w" signing_key_seq_num: 0 } } request_id { client_id: "93457ca4cd664565a2bb96397574b111" seq_no: 173 first_incomplete_seq_no: 173 attempt_no: 0 }
I20240503 07:59:23.755321 4286 raft_consensus.cc:2258] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.756495 4286 ts_tablet_manager.cc:1875] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:23.758534 4286 ts_tablet_manager.cc:1888] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.64
I20240503 07:59:23.758667 4286 log.cc:1196] T ebe8af8c7edf45368869b33dc1819e35 P cd82dbc60d0f40688fbe8e712dc7107a: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ebe8af8c7edf45368869b33dc1819e35
I20240503 07:59:23.759239 2251 catalog_manager.cc:4830] TS cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251): tablet ebe8af8c7edf45368869b33dc1819e35 (table rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]) successfully deleted
I20240503 07:59:23.761725 2536 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:23.763792 3689 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:23.765663 2801 raft_consensus.cc:2932] T df62f8897f4f4e4abe13f3455a665031 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 70 to 71, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.765772 2670 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 59 to 65, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.768193 3847 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 59 to 65, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.781730 3064 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 59 to 65, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.782662 2536 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 59 to 65, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.784034 4287 tablet_replica.cc:330] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: stopping tablet replica
I20240503 07:59:23.784373 4287 raft_consensus.cc:2229] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.784703 4287 pending_rounds.cc:62] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: Trying to abort 1 pending ops.
I20240503 07:59:23.791277 4266 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=57 ignored=0} inserts{seen=688 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:23.791298 4287 pending_rounds.cc:68] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: Aborting op as it isn't in flight: id { term: 1 index: 68 } timestamp: 7023506078176514048 op_type: WRITE_OP write_request { tablet_id: "df62f8897f4f4e4abe13f3455a665031" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\245D\325\001P\273~\275\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5bda9cf65aedd186" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506078066118656 authz_token { token_data: "\010\204\265\322\261\006\"3\n\005slave\022*\n b305a3a5cdc84cf28aa9445aa6b2879a\020\001\030\001 \001(\001" signature: "<redacted>""\210\2642\264g\377\203\202\265\262\262=qQ\337\364\344h\316\322\206\222m)m\332{\014\317\021\253\024\346\253l\023\006\007\010\2346\314\304\352=\312\272\017\330/q\001x\277\257\312d\340\027\313\337\255\332\303" signing_key_seq_num: 0 } } request_id { client_id: "cde0479281f64e8091d1d8bf1eb95d10" seq_no: 172 first_incomplete_seq_no: 172 attempt_no: 0 }
I20240503 07:59:23.791652 4266 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: Bootstrap complete.
I20240503 07:59:23.791728 4287 raft_consensus.cc:2258] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.791923 4266 ts_tablet_manager.cc:1367] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: Time spent bootstrapping tablet: real 0.257s user 0.055s sys 0.007s
I20240503 07:59:23.793545 4266 raft_consensus.cc:348] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: true } }
I20240503 07:59:23.793720 4266 raft_consensus.cc:729] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 886e4eb306064b98ab923623b8f1d401, State: Initialized, Role: LEARNER
I20240503 07:59:23.793855 4266 consensus_queue.cc:260] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: true } }
I20240503 07:59:23.793926 4287 ts_tablet_manager.cc:1875] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:23.794195 4266 ts_tablet_manager.cc:1398] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.794615 3084 tablet_copy_service.cc:342] P 81b9495629bb458892b031db80bc3715: Request end of tablet copy session 886e4eb306064b98ab923623b8f1d401-c9d779a081cd48628e24d1321f7b42af received from {username='slave'} at 127.0.99.131:60055
I20240503 07:59:23.794695 3084 tablet_copy_service.cc:433] P 81b9495629bb458892b031db80bc3715: ending tablet copy session 886e4eb306064b98ab923623b8f1d401-c9d779a081cd48628e24d1321f7b42af on tablet c9d779a081cd48628e24d1321f7b42af with peer 886e4eb306064b98ab923623b8f1d401
I20240503 07:59:23.796263 4287 ts_tablet_manager.cc:1888] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.68
I20240503 07:59:23.796929 3846 raft_consensus.cc:1203] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.44->[1.45-1.63] Dedup: 1.60->[1.61-1.63]
I20240503 07:59:23.801510 4287 log.cc:1196] T df62f8897f4f4e4abe13f3455a665031 P 886e4eb306064b98ab923623b8f1d401: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/df62f8897f4f4e4abe13f3455a665031
I20240503 07:59:23.802336 2253 catalog_manager.cc:4830] TS 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343): tablet df62f8897f4f4e4abe13f3455a665031 (table rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]) successfully deleted
I20240503 07:59:23.813655 4002 raft_consensus.cc:1050] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:23.814116 4002 consensus_queue.cc:237] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:23.821143 2409 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:23.822016 3846 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:23.822252 2932 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:23.822343 3503 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:23.822806 4085 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:23.823695 3503 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:23.832178 3532 raft_consensus.cc:1050] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:23.833093 3532 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:23.835084 2932 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:23.835775 3846 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:23.835705 3503 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index 45 to 65, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.836658 4166 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:23.836925 4166 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:23.837998 3064 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:23.838488 4136 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:23.838991 2259 catalog_manager.cc:5484] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index 45 to 65, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:23.841331 3059 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:23.843005 3987 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index 47 to 64, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.844707 2931 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 47 to 64, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.845187 3058 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 47 to 64, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.847594 2265 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 47 to 64, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.847914 3846 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 47 to 64, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.851817 2409 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 45 to 65, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.852852 2671 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240503 07:59:23.854457 2799 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:23.857529 4002 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:23.860242 3532 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:23.864316 2931 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 45 to 65, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.869153 3846 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 45 to 65, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.871783 2537 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:23.874408 3532 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:23.876319 2799 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 47 to 64, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.876401 3058 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 45 to 65, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.903512 4001 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240503 07:59:23.905942 2537 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 47 to 64, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:23.912326 2670 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 45 to 65, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
W20240503 07:59:23.957746 2474 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.133:46695, user_credentials={real_user=slave}} blocked reactor thread for 46047.3us
I20240503 07:59:23.989966 2671 raft_consensus.cc:1203] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.49->[1.50-1.63] Dedup: 1.57->[1.58-1.63]
I20240503 07:59:23.996150 2536 consensus_queue.cc:237] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:24.000358 2408 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240503 07:59:24.009264 4301 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 07:59:24.014118 3847 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240503 07:59:24.018679 2932 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240503 07:59:24.019042 2669 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240503 07:59:24.031816 4300 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 07:59:24.038553 4227 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=43 ignored=0} inserts{seen=715 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:59:24.038998 4227 tablet_bootstrap.cc:492] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:24.039283 4227 ts_tablet_manager.cc:1367] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.747s user 0.047s sys 0.008s
I20240503 07:59:24.039904 4227 raft_consensus.cc:348] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:24.040589 4227 raft_consensus.cc:729] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:24.040756 4227 consensus_queue.cc:260] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 1.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:24.041210 4227 ts_tablet_manager.cc:1398] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:24.041711 2820 tablet_copy_service.cc:342] P 0bc088d2657a49b39cb7981f0e3eec21: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-6a11003820e14dc7b2615f20baf179cc received from {username='slave'} at 127.0.99.135:57215
I20240503 07:59:24.041790 2820 tablet_copy_service.cc:433] P 0bc088d2657a49b39cb7981f0e3eec21: ending tablet copy session 6338e74c39eb4583842e387e64fce522-6a11003820e14dc7b2615f20baf179cc on tablet 6a11003820e14dc7b2615f20baf179cc with peer 6338e74c39eb4583842e387e64fce522
I20240503 07:59:24.064034 3351 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 07:59:24.069718 4002 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 07:59:24.088655 3503 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index 65 to 67, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.090960 2251 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d732123aab240c4b4216f451e87de09 with cas_config_opid_index 65: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:24.091657 2259 catalog_manager.cc:5484] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index 65 to 67, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:24.095551 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d732123aab240c4b4216f451e87de09 with delete_type TABLET_DATA_TOMBSTONED (TS 81b9495629bb458892b031db80bc3715 not found in new config with opid_index 67) from {username='slave'} at 127.0.0.1:37648
I20240503 07:59:24.100338 4310 tablet_replica.cc:330] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: stopping tablet replica
I20240503 07:59:24.102769 4310 raft_consensus.cc:2229] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.103304 4310 pending_rounds.cc:62] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: Trying to abort 1 pending ops.
I20240503 07:59:24.104524 4310 pending_rounds.cc:68] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: Aborting op as it isn't in flight: id { term: 1 index: 66 } timestamp: 7023506079689637888 op_type: WRITE_OP write_request { tablet_id: "7d732123aab240c4b4216f451e87de09" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000M\361X]V\365\013\232\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5a14444ed121162b" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506079392358400 authz_token { token_data: "\010\204\265\322\261\006\"3\n\005slave\022*\n b305a3a5cdc84cf28aa9445aa6b2879a\020\001\030\001 \001(\001" signature: "<redacted>""\210\2642\264g\377\203\202\265\262\262=qQ\337\364\344h\316\322\206\222m)m\332{\014\317\021\253\024\346\253l\023\006\007\010\2346\314\304\352=\312\272\017\330/q\001x\277\257\312d\340\027\313\337\255\332\303" signing_key_seq_num: 0 } } request_id { client_id: "cde0479281f64e8091d1d8bf1eb95d10" seq_no: 184 first_incomplete_seq_no: 184 attempt_no: 0 }
I20240503 07:59:24.105154 4310 raft_consensus.cc:2258] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.106120 2407 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index 65 to 67, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.107283 4310 ts_tablet_manager.cc:1875] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:24.142982 4310 ts_tablet_manager.cc:1888] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.66
I20240503 07:59:24.143091 4310 log.cc:1196] T 7d732123aab240c4b4216f451e87de09 P 81b9495629bb458892b031db80bc3715: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7d732123aab240c4b4216f451e87de09
I20240503 07:59:24.144017 2253 catalog_manager.cc:4830] TS 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145): tablet 7d732123aab240c4b4216f451e87de09 (table rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]) successfully deleted
I20240503 07:59:24.152031 2670 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index 65 to 67, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.157238 3847 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index 65 to 67, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.163218 2932 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index 65 to 67, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.170468 2669 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:24.172960 3846 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 07:59:24.174868 3987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 07:59:24.178848 2536 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 07:59:24.179153 2932 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 07:59:24.179658 3987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 07:59:24.180516 3987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 07:59:24.183733 4136 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 64 to 66, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.185937 2253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a4acef14f154d0b8fb94428a45a380b with cas_config_opid_index 64: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:24.188994 2259 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 reported cstate change: config changed from index 64 to 66, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:24.194558 2781 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a4acef14f154d0b8fb94428a45a380b with delete_type TABLET_DATA_TOMBSTONED (TS 0bc088d2657a49b39cb7981f0e3eec21 not found in new config with opid_index 66) from {username='slave'} at 127.0.0.1:39912
I20240503 07:59:24.194993 4316 tablet_replica.cc:330] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21: stopping tablet replica
I20240503 07:59:24.195394 4316 raft_consensus.cc:2229] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.195566 4316 raft_consensus.cc:2258] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.196591 3847 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 64 to 66, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.197299 3063 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240503 07:59:24.198993 3847 raft_consensus.cc:1203] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.36->[1.37-1.60] Dedup: 1.44->[1.45-1.60]
I20240503 07:59:24.199807 4316 ts_tablet_manager.cc:1875] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:24.200958 3971 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240503 07:59:24.202410 4316 ts_tablet_manager.cc:1888] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.65
I20240503 07:59:24.202574 4316 log.cc:1196] T 0a4acef14f154d0b8fb94428a45a380b P 0bc088d2657a49b39cb7981f0e3eec21: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:24.203130 3064 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 64 to 66, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.205138 2254 catalog_manager.cc:4830] TS 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199): tablet 0a4acef14f154d0b8fb94428a45a380b (table rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]) successfully deleted
I20240503 07:59:24.207865 2536 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 64 to 66, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.210801 2932 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 64 to 66, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.242637 3933 raft_consensus.cc:1050] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715: attempting to promote NON_VOTER 886e4eb306064b98ab923623b8f1d401 to VOTER
I20240503 07:59:24.243093 3933 consensus_queue.cc:237] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } }
I20240503 07:59:24.244838 2799 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 07:59:24.245190 2409 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 07:59:24.245671 4143 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 07:59:24.245728 2537 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 07:59:24.246114 4143 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 07:59:24.247049 4143 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 07:59:24.248667 2670 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 LEARNER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 07:59:24.248750 4143 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index 50 to 69, 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.249289 3932 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 07:59:24.251132 2670 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 50 to 69, 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.251617 2536 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 50 to 69, 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.251821 2408 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 50 to 69, 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.253839 2799 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 50 to 69, 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.254626 2265 catalog_manager.cc:5484] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index 50 to 69, 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.257743 2930 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240503 07:59:24.260198 4143 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20240503 07:59:24.261521 2930 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index 50 to 69, 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.268580 4308 raft_consensus.cc:1050] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:24.268983 4308 consensus_queue.cc:237] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:24.270462 2409 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 07:59:24.270795 3755 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 07:59:24.271514 3847 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 07:59:24.271644 2667 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 07:59:24.271893 2930 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 07:59:24.272632 3689 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 07:59:24.272959 2409 raft_consensus.cc:1203] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.62] Dedup: 1.61->[1.62-1.62]
I20240503 07:59:24.274542 3689 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 07:59:24.274655 4309 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 07:59:24.280817 4309 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index 37 to 62, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.281742 3847 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 37 to 62, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.283861 2407 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 37 to 62, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.284528 2670 raft_consensus.cc:1203] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.62] Dedup: 1.61->[1.62-1.62]
I20240503 07:59:24.284610 2267 catalog_manager.cc:5484] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522 reported cstate change: config changed from index 37 to 62, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.292640 2932 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 37 to 62, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.292428 2667 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 37 to 62, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
W20240503 07:59:24.325174 2474 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.133:46695, user_credentials={real_user=slave}} blocked reactor thread for 47488.7us
I20240503 07:59:24.327399 2537 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 07:59:24.345301 4176 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 57, Time since last communication: 0.000s
I20240503 07:59:24.350510 3058 consensus_queue.cc:237] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } }
I20240503 07:59:24.352128 2536 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 37 to 62, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.353649 2799 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 07:59:24.353883 2669 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 07:59:24.354195 2408 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 07:59:24.354314 4143 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 07:59:24.354574 3932 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 07:59:24.354844 4143 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 07:59:24.356384 3932 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index 69 to 71, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.357180 2408 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 69 to 71, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.358655 2253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9d779a081cd48628e24d1321f7b42af with cas_config_opid_index 69: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:24.359838 2261 catalog_manager.cc:5484] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 69 to 71, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:24.360615 2799 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 69 to 71, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.363047 2667 raft_consensus.cc:1203] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.70] Dedup: 1.70->[]
I20240503 07:59:24.365778 2671 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 69 to 71, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.384667 2931 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240503 07:59:24.394176 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d779a081cd48628e24d1321f7b42af with delete_type TABLET_DATA_TOMBSTONED (TS 3631cd1b9b79487581e9d31178f671c3 not found in new config with opid_index 71) from {username='slave'} at 127.0.0.1:52904
I20240503 07:59:24.395402 4324 tablet_replica.cc:330] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: stopping tablet replica
I20240503 07:59:24.395942 4324 raft_consensus.cc:2229] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.396164 4324 pending_rounds.cc:62] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: Trying to abort 1 pending ops.
I20240503 07:59:24.396260 4324 pending_rounds.cc:68] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: Aborting op as it isn't in flight: id { term: 1 index: 70 } timestamp: 7023506081152131072 op_type: WRITE_OP write_request { tablet_id: "c9d779a081cd48628e24d1321f7b42af" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\253\346\177ps+it\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""be25f8105d8204a0" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506081015234560 authz_token { token_data: "\010\204\265\322\261\006\"3\n\005slave\022*\n ca2a48a109274416a6bf71098168e532\020\001\030\001 \001(\001" signature: "<redacted>""-\005v\202\200\316L\356\303\321\217\325OUi:\305\377\336\341\225\305a \330\302C\374)\352\337\010\307C\030\014.\250\243<\003J\031\t$\352\370\230!\200\364)6\274\336J\357\230\2202{\300\330\030" signing_key_seq_num: 0 } } request_id { client_id: "579b2a7f70774018b4ac5ad0aa06fbe2" seq_no: 179 first_incomplete_seq_no: 179 attempt_no: 0 }
I20240503 07:59:24.397883 4143 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240503 07:59:24.398253 4324 raft_consensus.cc:2258] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.399428 2931 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 69 to 71, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false } } }
I20240503 07:59:24.410879 4324 ts_tablet_manager.cc:1875] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:24.423879 4324 ts_tablet_manager.cc:1888] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.70
I20240503 07:59:24.424050 4324 log.cc:1196] T c9d779a081cd48628e24d1321f7b42af P 3631cd1b9b79487581e9d31178f671c3: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c9d779a081cd48628e24d1321f7b42af
I20240503 07:59:24.426440 2251 catalog_manager.cc:4830] TS 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355): tablet c9d779a081cd48628e24d1321f7b42af (table rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]) successfully deleted
I20240503 07:59:24.436388 2799 consensus_queue.cc:237] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:24.441769 2407 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:24.442174 3689 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:24.444401 3847 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:24.445518 4176 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:24.448782 2536 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:24.451048 3689 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index 62 to 64, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.451352 4176 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:24.452888 2254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a11003820e14dc7b2615f20baf179cc with cas_config_opid_index 62: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:24.452994 2407 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 62 to 64, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.454542 2261 catalog_manager.cc:5484] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: config changed from index 62 to 64, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:24.456105 3847 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 62 to 64, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.459481 2911 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a11003820e14dc7b2615f20baf179cc with delete_type TABLET_DATA_TOMBSTONED (TS e3b61927599645d4b11278833b2f27d6 not found in new config with opid_index 64) from {username='slave'} at 127.0.0.1:36192
I20240503 07:59:24.459973 2669 raft_consensus.cc:1261] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240503 07:59:24.461695 3755 consensus_queue.cc:1035] T 6a11003820e14dc7b2615f20baf179cc P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 07:59:24.461848 2408 raft_consensus.cc:1203] T 6a11003820e14dc7b2615f20baf179cc P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63] Dedup: 1.63->[]
I20240503 07:59:24.466008 2669 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 62 to 64, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.476971 2537 raft_consensus.cc:2932] T 6a11003820e14dc7b2615f20baf179cc P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 62 to 64, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:24.550160 4331 tablet_replica.cc:330] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: stopping tablet replica
I20240503 07:59:24.553741 4331 raft_consensus.cc:2229] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.553978 4331 pending_rounds.cc:62] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: Trying to abort 1 pending ops.
I20240503 07:59:24.554404 4331 pending_rounds.cc:68] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: Aborting op as it isn't in flight: id { term: 1 index: 63 } timestamp: 7023506081494118400 op_type: WRITE_OP write_request { tablet_id: "6a11003820e14dc7b2615f20baf179cc" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\032\177\203\177\355e\030\306\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""c8b301f6408b83e0" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506081431126016 authz_token { token_data: "\010\204\265\322\261\006\"3\n\005slave\022*\n ca2a48a109274416a6bf71098168e532\020\001\030\001 \001(\001" signature: "<redacted>""-\005v\202\200\316L\356\303\321\217\325OUi:\305\377\336\341\225\305a \330\302C\374)\352\337\010\307C\030\014.\250\243<\003J\031\t$\352\370\230!\200\364)6\274\336J\357\230\2202{\300\330\030" signing_key_seq_num: 0 } } request_id { client_id: "579b2a7f70774018b4ac5ad0aa06fbe2" seq_no: 183 first_incomplete_seq_no: 183 attempt_no: 0 }
I20240503 07:59:24.554819 4331 raft_consensus.cc:2258] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.589300 4331 ts_tablet_manager.cc:1875] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:24.599153 4331 ts_tablet_manager.cc:1888] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.63
I20240503 07:59:24.600524 4331 log.cc:1196] T 6a11003820e14dc7b2615f20baf179cc P e3b61927599645d4b11278833b2f27d6: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/6a11003820e14dc7b2615f20baf179cc
I20240503 07:59:24.603615 2252 catalog_manager.cc:4830] TS e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695): tablet 6a11003820e14dc7b2615f20baf179cc (table rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]) successfully deleted
I20240503 07:59:24.912850 2537 consensus_queue.cc:237] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:24.914721 2802 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240503 07:59:24.915072 3438 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240503 07:59:24.916672 2930 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
W20240503 07:59:24.918584 2474 consensus_peers.cc:485] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccac60c83bd54650a50a07d87e3fc72a. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:24.922032 3503 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240503 07:59:24.924077 2407 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240503 07:59:24.927594 4300 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index -1 to 95, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:24.928154 2931 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index -1 to 95, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:24.932677 2259 catalog_manager.cc:5484] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 reported cstate change: config changed from index -1 to 95, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:24.938926 2671 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240503 07:59:24.939355 3919 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240503 07:59:24.940487 2802 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index -1 to 95, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:24.960892 3503 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240503 07:59:24.983995 2407 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index -1 to 95, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:24.987172 2670 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index -1 to 95, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:25.089001 2540 consensus_queue.cc:237] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:25.090893 2798 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:25.091342 3503 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 07:59:25.099503 2408 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:25.105191 3064 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:25.105738 3351 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 07:59:25.114084 2929 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:25.117478 3958 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 07:59:25.120340 3503 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:25.123881 2408 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:25.124570 2259 catalog_manager.cc:5484] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:25.130120 3059 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
W20240503 07:59:25.146441 2474 consensus_peers.cc:485] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbbc403671d24146b48f8e0c14db7de6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:25.146677 3958 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 74, Time since last communication: 0.000s
I20240503 07:59:25.162652 2931 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:25.170385 2802 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:25.172461 2931 consensus_queue.cc:237] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:25.174319 3064 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:25.174731 3654 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 07:59:25.177935 2540 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:25.179699 4084 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 07:59:25.180128 2801 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:25.180711 4084 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 07:59:25.182950 4084 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:25.185039 2801 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:25.185317 2540 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:25.187170 2670 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 07:59:25.189280 3064 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:25.189427 4018 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 07:59:25.189882 2267 catalog_manager.cc:5484] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 reported cstate change: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "e3b61927599645d4b11278833b2f27d6" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:25.213554 2671 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
W20240503 07:59:25.281565 2867 consensus_peers.cc:485] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 813f904dbfab4d46964bbbf877715cc9. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:25.498041 4373 ts_tablet_manager.cc:902] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355)
I20240503 07:59:25.498585 4373 tablet_copy_client.cc:308] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40355
I20240503 07:59:25.505813 4384 ts_tablet_manager.cc:902] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355)
I20240503 07:59:25.506284 4384 tablet_copy_client.cc:308] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40355
I20240503 07:59:25.517261 2560 tablet_copy_service.cc:140] P 3631cd1b9b79487581e9d31178f671c3: Received BeginTabletCopySession request for tablet ccac60c83bd54650a50a07d87e3fc72a from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:58999)
I20240503 07:59:25.517356 2560 tablet_copy_service.cc:161] P 3631cd1b9b79487581e9d31178f671c3: Beginning new tablet copy session on tablet ccac60c83bd54650a50a07d87e3fc72a from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:58999: session id = 6338e74c39eb4583842e387e64fce522-ccac60c83bd54650a50a07d87e3fc72a
I20240503 07:59:25.518306 2560 tablet_copy_source_session.cc:205] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:25.518965 2560 tablet_copy_service.cc:140] P 3631cd1b9b79487581e9d31178f671c3: Received BeginTabletCopySession request for tablet dbbc403671d24146b48f8e0c14db7de6 from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:58999)
I20240503 07:59:25.519045 2560 tablet_copy_service.cc:161] P 3631cd1b9b79487581e9d31178f671c3: Beginning new tablet copy session on tablet dbbc403671d24146b48f8e0c14db7de6 from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:58999: session id = 6338e74c39eb4583842e387e64fce522-dbbc403671d24146b48f8e0c14db7de6
I20240503 07:59:25.519744 2560 tablet_copy_source_session.cc:205] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:25.527134 4373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccac60c83bd54650a50a07d87e3fc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.567287 4384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbc403671d24146b48f8e0c14db7de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.684288 4373 tablet_copy_client.cc:783] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:25.684473 4373 tablet_copy_client.cc:647] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:25.715512 4384 tablet_copy_client.cc:783] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:25.724273 4384 tablet_copy_client.cc:647] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:25.746412 4384 tablet_copy_client.cc:523] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:25.786306 4384 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:25.794703 4373 tablet_copy_client.cc:523] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:25.807641 4373 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:25.815599 4394 ts_tablet_manager.cc:902] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:25.816184 4394 tablet_copy_client.cc:308] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46695
I20240503 07:59:25.859829 2953 tablet_copy_service.cc:140] P e3b61927599645d4b11278833b2f27d6: Received BeginTabletCopySession request for tablet 813f904dbfab4d46964bbbf877715cc9 from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:43891)
I20240503 07:59:25.859972 2953 tablet_copy_service.cc:161] P e3b61927599645d4b11278833b2f27d6: Beginning new tablet copy session on tablet 813f904dbfab4d46964bbbf877715cc9 from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:43891: session id = 6338e74c39eb4583842e387e64fce522-813f904dbfab4d46964bbbf877715cc9
I20240503 07:59:25.861469 2953 tablet_copy_source_session.cc:205] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:25.867586 4394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f904dbfab4d46964bbbf877715cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.871487 4394 tablet_copy_client.cc:783] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:25.880785 4394 tablet_copy_client.cc:647] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:25.884176 4394 tablet_copy_client.cc:523] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:25.885902 4394 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:25.998584 4394 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=773 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:25.998973 4394 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:25.999256 4394 ts_tablet_manager.cc:1367] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.113s user 0.075s sys 0.007s
I20240503 07:59:26.000010 4394 raft_consensus.cc:348] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:26.000241 4394 raft_consensus.cc:729] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:26.000393 4394 consensus_queue.cc:260] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:26.000869 4394 ts_tablet_manager.cc:1398] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.001833 2953 tablet_copy_service.cc:342] P e3b61927599645d4b11278833b2f27d6: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-813f904dbfab4d46964bbbf877715cc9 received from {username='slave'} at 127.0.99.135:43891
I20240503 07:59:26.001925 2953 tablet_copy_service.cc:433] P e3b61927599645d4b11278833b2f27d6: ending tablet copy session 6338e74c39eb4583842e387e64fce522-813f904dbfab4d46964bbbf877715cc9 on tablet 813f904dbfab4d46964bbbf877715cc9 with peer 6338e74c39eb4583842e387e64fce522
I20240503 07:59:26.081682 4384 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=81 ignored=0} inserts{seen=766 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.082183 4384 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:26.082399 4384 ts_tablet_manager.cc:1367] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.296s user 0.060s sys 0.016s
I20240503 07:59:26.082859 4384 raft_consensus.cc:348] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:26.082996 4384 raft_consensus.cc:729] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:26.083086 4384 consensus_queue.cc:260] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 81, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:26.083344 4384 ts_tablet_manager.cc:1398] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.084502 2560 tablet_copy_service.cc:342] P 3631cd1b9b79487581e9d31178f671c3: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-dbbc403671d24146b48f8e0c14db7de6 received from {username='slave'} at 127.0.99.135:58999
I20240503 07:59:26.084589 2560 tablet_copy_service.cc:433] P 3631cd1b9b79487581e9d31178f671c3: ending tablet copy session 6338e74c39eb4583842e387e64fce522-dbbc403671d24146b48f8e0c14db7de6 on tablet dbbc403671d24146b48f8e0c14db7de6 with peer 6338e74c39eb4583842e387e64fce522
I20240503 07:59:26.125130 3846 raft_consensus.cc:1203] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.75->[1.76-1.89] Dedup: 1.88->[1.89-1.89]
I20240503 07:59:26.134316 4373 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=107 overwritten=0 applied=107 ignored=0} inserts{seen=792 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:26.134634 4373 tablet_bootstrap.cc:492] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:26.134825 4373 ts_tablet_manager.cc:1367] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.327s user 0.090s sys 0.004s
I20240503 07:59:26.135253 4373 raft_consensus.cc:348] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:26.135378 4373 raft_consensus.cc:729] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:26.135823 4373 consensus_queue.cc:260] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:26.136216 4373 ts_tablet_manager.cc:1398] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.136689 2560 tablet_copy_service.cc:342] P 3631cd1b9b79487581e9d31178f671c3: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-ccac60c83bd54650a50a07d87e3fc72a received from {username='slave'} at 127.0.99.135:58999
I20240503 07:59:26.136773 2560 tablet_copy_service.cc:433] P 3631cd1b9b79487581e9d31178f671c3: ending tablet copy session 6338e74c39eb4583842e387e64fce522-ccac60c83bd54650a50a07d87e3fc72a on tablet ccac60c83bd54650a50a07d87e3fc72a with peer 6338e74c39eb4583842e387e64fce522
W20240503 07:59:26.148078 2474 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.133:46695, user_credentials={real_user=slave}} blocked reactor thread for 75865.6us
I20240503 07:59:26.164340 3846 raft_consensus.cc:1203] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.75->[1.76-1.96] Dedup: 1.81->[1.82-1.96]
I20240503 07:59:26.330498 3654 raft_consensus.cc:1050] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:26.330997 3654 consensus_queue.cc:237] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:26.333662 3059 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240503 07:59:26.334129 2800 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240503 07:59:26.334228 3847 raft_consensus.cc:1203] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.94->[1.95-1.124] Dedup: 1.107->[1.108-1.124]
I20240503 07:59:26.334451 4018 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240503 07:59:26.334720 4018 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240503 07:59:26.337163 2536 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240503 07:59:26.337924 4017 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240503 07:59:26.339870 3848 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240503 07:59:26.340694 4017 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240503 07:59:26.341745 4017 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index 76 to 91, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.344416 2261 catalog_manager.cc:5484] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 reported cstate change: config changed from index 76 to 91, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3b61927599645d4b11278833b2f27d6" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.345510 2536 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 76 to 91, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.346069 3059 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 76 to 91, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.347343 2669 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240503 07:59:26.352046 2800 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 76 to 91, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.352227 3848 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 76 to 91, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.357836 4018 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240503 07:59:26.384507 2667 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 76 to 91, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.393170 2536 consensus_queue.cc:237] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 1.130, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:26.399617 2667 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240503 07:59:26.399842 2409 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240503 07:59:26.400245 4301 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240503 07:59:26.421535 4301 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240503 07:59:26.422003 2931 consensus_queue.cc:237] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:26.424530 3059 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240503 07:59:26.427742 4415 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240503 07:59:26.433791 3958 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:26.434772 3846 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240503 07:59:26.435132 3654 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 07:59:26.435801 2407 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:26.436959 2265 catalog_manager.cc:5484] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index -1 to 131, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.438969 3063 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240503 07:59:26.439307 3654 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 07:59:26.440282 2800 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240503 07:59:26.440670 3654 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.442402 3654 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 07:59:26.442790 2252 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 813f904dbfab4d46964bbbf877715cc9 with cas_config_opid_index 91: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:26.443130 3063 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.443238 3846 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.443898 2800 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.445932 2669 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240503 07:59:26.446563 2265 catalog_manager.cc:5484] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 91 to 93, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "e3b61927599645d4b11278833b2f27d6" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.448308 2669 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:26.449342 4368 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 07:59:26.450423 3063 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
W20240503 07:59:26.450700 2474 consensus_peers.cc:485] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71eeb5740a7344749b534273ba1c9922. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.472373 2931 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240503 07:59:26.477605 2667 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.477839 3503 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240503 07:59:26.479913 3958 raft_consensus.cc:1050] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:26.480399 3958 consensus_queue.cc:237] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.124, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:26.483063 2406 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240503 07:59:26.483121 2801 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240503 07:59:26.484980 3438 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240503 07:59:26.485162 3503 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240503 07:59:26.486887 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet 813f904dbfab4d46964bbbf877715cc9 with delete_type TABLET_DATA_TOMBSTONED (TS 3631cd1b9b79487581e9d31178f671c3 not found in new config with opid_index 93) from {username='slave'} at 127.0.0.1:52904
I20240503 07:59:26.488837 2671 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240503 07:59:26.499228 3438 raft_consensus.cc:1050] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:26.499588 3438 consensus_queue.cc:237] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:26.501549 2406 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 07:59:26.501982 3503 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 07:59:26.502406 2797 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 07:59:26.503340 3503 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 07:59:26.504874 2932 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index -1 to 131, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:26.505153 2929 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240503 07:59:26.507809 3848 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240503 07:59:26.509581 4425 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240503 07:59:26.509678 3847 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 07:59:26.509932 2932 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 07:59:26.511072 3958 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240503 07:59:26.518020 4415 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240503 07:59:26.520221 4415 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 07:59:26.520532 4466 tablet_replica.cc:330] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3: stopping tablet replica
I20240503 07:59:26.520819 4466 raft_consensus.cc:2229] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.521245 4466 raft_consensus.cc:2258] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.524171 4415 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 07:59:26.525277 4415 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.125: config changed from index 95 to 125, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.528988 3846 raft_consensus.cc:1203] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.103->[1.104-1.105] Dedup: 1.104->[1.105-1.105]
I20240503 07:59:26.531538 2266 catalog_manager.cc:5484] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index 95 to 125, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:26.533147 2666 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 95 to 125, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.533149 2406 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 95 to 125, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.540009 3846 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 95 to 125, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.540171 2797 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 95 to 125, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.540625 3958 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 76 to 105, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.542658 2409 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 76 to 105, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.544380 3059 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 07:59:26.547529 2267 catalog_manager.cc:5484] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index 76 to 105, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.547967 3846 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 76 to 105, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.548981 2931 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 95 to 125, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.560711 2932 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 76 to 105, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.572270 2536 consensus_queue.cc:237] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.127, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:26.546257 2801 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 76 to 105, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.587672 4425 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 07:59:26.590382 3063 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 76 to 105, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.590698 2666 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240503 07:59:26.597632 4425 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 125, Time since last communication: 0.000s
I20240503 07:59:26.599256 2407 consensus_queue.cc:237] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 1.125, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:26.603099 2536 consensus_queue.cc:237] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:26.603303 3846 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240503 07:59:26.605589 2932 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240503 07:59:26.607023 3059 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 07:59:26.607353 4425 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 07:59:26.605301 2409 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 07:59:26.608146 4425 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 07:59:26.609956 3846 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 07:59:26.609966 2932 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240503 07:59:26.611140 3503 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 07:59:26.612382 4466 ts_tablet_manager.cc:1875] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:26.613155 2801 raft_consensus.cc:1261] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240503 07:59:26.614252 2667 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 07:59:26.614567 3060 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 07:59:26.614554 3392 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240503 07:59:26.614605 2932 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 07:59:26.614919 3958 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 07:59:26.615022 3696 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240503 07:59:26.615309 3696 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240503 07:59:26.615255 3438 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 07:59:26.616380 3958 consensus_queue.cc:1035] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 126, Time since last communication: 0.000s
I20240503 07:59:26.617345 3392 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEADER]: Committing config change with OpId 1.126: config changed from index -1 to 126, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:26.618156 3503 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 07:59:26.619206 2932 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index -1 to 126, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:26.619758 2667 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index -1 to 126, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:26.620460 2267 catalog_manager.cc:5484] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index -1 to 126, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" committed_config { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.621737 3060 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index -1 to 126, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:26.624675 2801 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240503 07:59:26.625272 3730 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 124, Time since last communication: 0.000s
I20240503 07:59:26.625978 3060 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 105 to 108, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.626252 3438 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index 105 to 108, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.627547 3438 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.128: config changed from index 125 to 128, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.627705 2671 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index 125 to 128, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.629618 2261 catalog_manager.cc:5484] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index 105 to 108, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.631987 2266 catalog_manager.cc:5484] T ccac60c83bd54650a50a07d87e3fc72a P 886e4eb306064b98ab923623b8f1d401 reported cstate change: config changed from index 125 to 128, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.632257 2802 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index 125 to 128, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.633903 2406 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 105 to 108, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.636138 2389 tablet_service.cc:1514] Processing DeleteTablet for tablet ccac60c83bd54650a50a07d87e3fc72a with delete_type TABLET_DATA_TOMBSTONED (TS cd82dbc60d0f40688fbe8e712dc7107a not found in new config with opid_index 128) from {username='slave'} at 127.0.0.1:59282
I20240503 07:59:26.636427 2781 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbc403671d24146b48f8e0c14db7de6 with delete_type TABLET_DATA_TOMBSTONED (TS 0bc088d2657a49b39cb7981f0e3eec21 not found in new config with opid_index 108) from {username='slave'} at 127.0.0.1:39912
I20240503 07:59:26.641129 2251 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dbbc403671d24146b48f8e0c14db7de6 with cas_config_opid_index 105: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:26.641994 2251 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccac60c83bd54650a50a07d87e3fc72a with cas_config_opid_index 125: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:26.644328 4466 ts_tablet_manager.cc:1888] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.92
I20240503 07:59:26.644400 4466 log.cc:1196] T 813f904dbfab4d46964bbbf877715cc9 P 3631cd1b9b79487581e9d31178f671c3: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/813f904dbfab4d46964bbbf877715cc9
I20240503 07:59:26.644677 4474 tablet_replica.cc:330] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: stopping tablet replica
I20240503 07:59:26.645210 4475 tablet_replica.cc:330] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: stopping tablet replica
I20240503 07:59:26.647717 2251 catalog_manager.cc:4830] TS 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355): tablet 813f904dbfab4d46964bbbf877715cc9 (table rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]) successfully deleted
I20240503 07:59:26.648451 4474 raft_consensus.cc:2229] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.648633 4475 raft_consensus.cc:2229] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.647414 2801 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index -1 to 126, NON_VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) added. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } } }
I20240503 07:59:26.649047 4474 pending_rounds.cc:62] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: Trying to abort 1 pending ops.
I20240503 07:59:26.649160 4475 pending_rounds.cc:62] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: Trying to abort 1 pending ops.
I20240503 07:59:26.649253 4475 pending_rounds.cc:68] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: Aborting op as it isn't in flight: id { term: 1 index: 107 } timestamp: 7023506090390458368 op_type: WRITE_OP write_request { tablet_id: "dbbc403671d24146b48f8e0c14db7de6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\341\227w3r\013\332\030\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""0036b9de91c97d83" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506090308259840 authz_token { token_data: "\010\204\265\322\261\006\"3\n\005slave\022*\n ca2a48a109274416a6bf71098168e532\020\001\030\001 \001(\001" signature: "<redacted>""-\005v\202\200\316L\356\303\321\217\325OUi:\305\377\336\341\225\305a \330\302C\374)\352\337\010\307C\030\014.\250\243<\003J\031\t$\352\370\230!\200\364)6\274\336J\357\230\2202{\300\330\030" signing_key_seq_num: 0 } } request_id { client_id: "579b2a7f70774018b4ac5ad0aa06fbe2" seq_no: 325 first_incomplete_seq_no: 325 attempt_no: 0 }
I20240503 07:59:26.649646 4474 pending_rounds.cc:68] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: Aborting op as it isn't in flight: id { term: 1 index: 127 } timestamp: 7023506090146488320 op_type: WRITE_OP write_request { tablet_id: "ccac60c83bd54650a50a07d87e3fc72a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\034\031\027=?\035\340\203\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a7e427386795c3fb" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506090063527936 authz_token { token_data: "\010\204\265\322\261\006\"3\n\005slave\022*\n 47bd34c090d04940ba0ab84776e9fabb\020\001\030\001 \001(\001" signature: "<redacted>""?\024\213\243b?s\323\361~\203X\026\r\241V\257\037\332\005\006Xf\231\213\276v\025\342\304\253\343;_zn\230\301\001\360\014\323\327ics)S\235\035\014\217d\271\322\325\263[L\307QT\026P" signing_key_seq_num: 0 } } request_id { client_id: "92f9dc00bf774380a0d18a000d1e0552" seq_no: 383 first_incomplete_seq_no: 383 attempt_no: 0 }
I20240503 07:59:26.649974 4475 raft_consensus.cc:2258] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.650259 4474 raft_consensus.cc:2258] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.677090 4475 ts_tablet_manager.cc:1875] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:26.677721 4474 ts_tablet_manager.cc:1875] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:26.681262 4475 ts_tablet_manager.cc:1888] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.107
I20240503 07:59:26.682098 4474 ts_tablet_manager.cc:1888] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.127
I20240503 07:59:26.682302 4474 log.cc:1196] T ccac60c83bd54650a50a07d87e3fc72a P cd82dbc60d0f40688fbe8e712dc7107a: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ccac60c83bd54650a50a07d87e3fc72a
I20240503 07:59:26.682396 4475 log.cc:1196] T dbbc403671d24146b48f8e0c14db7de6 P 0bc088d2657a49b39cb7981f0e3eec21: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/dbbc403671d24146b48f8e0c14db7de6
I20240503 07:59:26.684775 2251 catalog_manager.cc:4830] TS cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251): tablet ccac60c83bd54650a50a07d87e3fc72a (table rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]) successfully deleted
I20240503 07:59:26.685231 2927 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index 125 to 128, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.686769 2931 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 105 to 108, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.689126 2254 catalog_manager.cc:4830] TS 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199): tablet dbbc403671d24146b48f8e0c14db7de6 (table rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]) successfully deleted
W20240503 07:59:26.703225 2343 consensus_peers.cc:485] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a -> Peer 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): Couldn't send request to peer 6338e74c39eb4583842e387e64fce522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d116d60ce3a04fc782762c4b8808a178. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:26.717315 3846 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 105 to 108, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.721787 3847 raft_consensus.cc:2932] T ccac60c83bd54650a50a07d87e3fc72a P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index 125 to 128, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:26.925354 4510 ts_tablet_manager.cc:902] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355)
I20240503 07:59:26.925863 4510 tablet_copy_client.cc:308] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40355
I20240503 07:59:26.926999 2560 tablet_copy_service.cc:140] P 3631cd1b9b79487581e9d31178f671c3: Received BeginTabletCopySession request for tablet 71eeb5740a7344749b534273ba1c9922 from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:58999)
I20240503 07:59:26.927107 2560 tablet_copy_service.cc:161] P 3631cd1b9b79487581e9d31178f671c3: Beginning new tablet copy session on tablet 71eeb5740a7344749b534273ba1c9922 from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:58999: session id = 6338e74c39eb4583842e387e64fce522-71eeb5740a7344749b534273ba1c9922
I20240503 07:59:26.927981 2560 tablet_copy_source_session.cc:205] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:26.929672 4510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71eeb5740a7344749b534273ba1c9922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.933728 4510 tablet_copy_client.cc:783] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:26.933954 4510 tablet_copy_client.cc:647] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:26.936458 4510 tablet_copy_client.cc:523] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:26.937633 4510 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:27.108987 4523 ts_tablet_manager.cc:902] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: Initiating tablet copy from peer cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251)
I20240503 07:59:27.112027 4523 tablet_copy_client.cc:308] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38251
I20240503 07:59:27.130028 2429 tablet_copy_service.cc:140] P cd82dbc60d0f40688fbe8e712dc7107a: Received BeginTabletCopySession request for tablet d116d60ce3a04fc782762c4b8808a178 from peer 6338e74c39eb4583842e387e64fce522 ({username='slave'} at 127.0.99.135:38041)
I20240503 07:59:27.130168 2429 tablet_copy_service.cc:161] P cd82dbc60d0f40688fbe8e712dc7107a: Beginning new tablet copy session on tablet d116d60ce3a04fc782762c4b8808a178 from peer 6338e74c39eb4583842e387e64fce522 at {username='slave'} at 127.0.99.135:38041: session id = 6338e74c39eb4583842e387e64fce522-d116d60ce3a04fc782762c4b8808a178
I20240503 07:59:27.131618 2429 tablet_copy_source_session.cc:205] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:27.133131 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116d60ce3a04fc782762c4b8808a178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.135324 4523 tablet_copy_client.cc:783] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:27.135453 4523 tablet_copy_client.cc:647] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:27.217270 4510 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=140 overwritten=0 applied=139 ignored=0} inserts{seen=811 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 07:59:27.217697 4510 tablet_bootstrap.cc:492] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:27.217981 4510 ts_tablet_manager.cc:1367] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.280s user 0.116s sys 0.002s
I20240503 07:59:27.254170 4510 raft_consensus.cc:348] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:27.254840 4510 raft_consensus.cc:729] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:27.254981 4510 consensus_queue.cc:260] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 139, Last appended: 1.140, Last appended by leader: 140, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:27.255337 4510 ts_tablet_manager.cc:1398] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.037s user 0.001s sys 0.000s
I20240503 07:59:27.257673 2560 tablet_copy_service.cc:342] P 3631cd1b9b79487581e9d31178f671c3: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-71eeb5740a7344749b534273ba1c9922 received from {username='slave'} at 127.0.99.135:58999
I20240503 07:59:27.257757 2560 tablet_copy_service.cc:433] P 3631cd1b9b79487581e9d31178f671c3: ending tablet copy session 6338e74c39eb4583842e387e64fce522-71eeb5740a7344749b534273ba1c9922 on tablet 71eeb5740a7344749b534273ba1c9922 with peer 6338e74c39eb4583842e387e64fce522
I20240503 07:59:27.304347 4523 tablet_copy_client.cc:523] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:27.305758 4523 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: Bootstrap starting.
I20240503 07:59:27.324752 3848 raft_consensus.cc:1203] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.130->[1.131-1.151] Dedup: 1.140->[1.141-1.151]
I20240503 07:59:27.506289 3503 raft_consensus.cc:1050] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:27.506664 3503 consensus_queue.cc:237] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 1.155, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:27.508447 2406 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240503 07:59:27.511441 3438 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240503 07:59:27.509986 2671 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240503 07:59:27.513249 2927 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240503 07:59:27.513690 3846 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240503 07:59:27.514135 3064 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240503 07:59:27.515779 3438 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 153, Time since last communication: 0.000s
I20240503 07:59:27.517282 4425 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240503 07:59:27.518369 3438 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240503 07:59:27.523833 4425 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240503 07:59:27.548986 3351 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.156: config changed from index 131 to 156, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.552964 2409 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 131 to 156, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.555672 2665 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 131 to 156, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.557255 2267 catalog_manager.cc:5484] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index 131 to 156, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.560269 3846 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 131 to 156, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.569242 2927 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 131 to 156, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.574811 3064 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 131 to 156, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.784984 2537 consensus_queue.cc:237] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 159, Committed index: 159, Last appended: 1.160, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:27.789139 3848 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 07:59:27.789157 2408 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 07:59:27.789485 3503 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 07:59:27.793382 4425 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 07:59:27.794062 2932 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 07:59:27.799563 3064 raft_consensus.cc:1261] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 1 index: 161. (index mismatch)
I20240503 07:59:27.801014 3503 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 07:59:27.805179 4301 consensus_queue.cc:1035] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 159, Time since last communication: 0.000s
I20240503 07:59:27.805570 3958 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.161: config changed from index 156 to 161, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.808526 2258 catalog_manager.cc:5484] T 71eeb5740a7344749b534273ba1c9922 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index 156 to 161, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:27.811012 2929 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 156 to 161, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.812978 2650 tablet_service.cc:1514] Processing DeleteTablet for tablet 71eeb5740a7344749b534273ba1c9922 with delete_type TABLET_DATA_TOMBSTONED (TS 886e4eb306064b98ab923623b8f1d401 not found in new config with opid_index 161) from {username='slave'} at 127.0.0.1:52688
I20240503 07:59:27.828190 2409 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 156 to 161, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.831344 3848 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 156 to 161, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.840559 2251 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71eeb5740a7344749b534273ba1c9922 with cas_config_opid_index 156: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:27.843794 3064 raft_consensus.cc:2932] T 71eeb5740a7344749b534273ba1c9922 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.161: config changed from index 156 to 161, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 161 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:27.856197 4556 tablet_replica.cc:330] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: stopping tablet replica
I20240503 07:59:27.856328 4556 raft_consensus.cc:2229] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.856426 4556 pending_rounds.cc:62] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: Trying to abort 1 pending ops.
I20240503 07:59:27.856454 4556 pending_rounds.cc:68] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: Aborting op as it isn't in flight: id { term: 1 index: 160 } timestamp: 7023506095161069568 op_type: WRITE_OP write_request { tablet_id: "71eeb5740a7344749b534273ba1c9922" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000{\267\230>\340sd \000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""512a4d9c609a8b23" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506094880550912 authz_token { token_data: "\010\204\265\322\261\006\"3\n\005slave\022*\n 47bd34c090d04940ba0ab84776e9fabb\020\001\030\001 \001(\001" signature: "<redacted>""?\024\213\243b?s\323\361~\203X\026\r\241V\257\037\332\005\006Xf\231\213\276v\025\342\304\253\343;_zn\230\301\001\360\014\323\327ics)S\235\035\014\217d\271\322\325\263[L\307QT\026P" signing_key_seq_num: 0 } } request_id { client_id: "92f9dc00bf774380a0d18a000d1e0552" seq_no: 434 first_incomplete_seq_no: 434 attempt_no: 0 }
I20240503 07:59:27.856819 4556 raft_consensus.cc:2258] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.857599 4556 ts_tablet_manager.cc:1875] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:27.860545 4556 ts_tablet_manager.cc:1888] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.160
I20240503 07:59:27.860622 4556 log.cc:1196] T 71eeb5740a7344749b534273ba1c9922 P 886e4eb306064b98ab923623b8f1d401: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/71eeb5740a7344749b534273ba1c9922
I20240503 07:59:27.861461 2253 catalog_manager.cc:4830] TS 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343): tablet 71eeb5740a7344749b534273ba1c9922 (table rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]) successfully deleted
I20240503 07:59:27.889714 4523 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: Bootstrap replayed 1/1 log segments. Stats: ops{read=142 overwritten=0 applied=142 ignored=0} inserts{seen=814 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:27.889999 4523 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: Bootstrap complete.
I20240503 07:59:27.893627 4523 ts_tablet_manager.cc:1367] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: Time spent bootstrapping tablet: real 0.588s user 0.120s sys 0.011s
I20240503 07:59:27.894098 4523 raft_consensus.cc:348] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:27.894234 4523 raft_consensus.cc:729] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6338e74c39eb4583842e387e64fce522, State: Initialized, Role: LEARNER
I20240503 07:59:27.894335 4523 consensus_queue.cc:260] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 142, Last appended: 1.142, Last appended by leader: 142, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: true } }
I20240503 07:59:27.895408 4523 ts_tablet_manager.cc:1398] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:27.897027 2429 tablet_copy_service.cc:342] P cd82dbc60d0f40688fbe8e712dc7107a: Request end of tablet copy session 6338e74c39eb4583842e387e64fce522-d116d60ce3a04fc782762c4b8808a178 received from {username='slave'} at 127.0.99.135:38041
I20240503 07:59:27.897146 2429 tablet_copy_service.cc:433] P cd82dbc60d0f40688fbe8e712dc7107a: ending tablet copy session 6338e74c39eb4583842e387e64fce522-d116d60ce3a04fc782762c4b8808a178 on tablet d116d60ce3a04fc782762c4b8808a178 with peer 6338e74c39eb4583842e387e64fce522
I20240503 07:59:28.086246 3846 raft_consensus.cc:1203] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.125->[1.126-1.164] Dedup: 1.142->[1.143-1.164]
I20240503 07:59:28.246971 4467 raft_consensus.cc:1050] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a: attempting to promote NON_VOTER 6338e74c39eb4583842e387e64fce522 to VOTER
I20240503 07:59:28.248288 4467 consensus_queue.cc:237] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 166, Committed index: 166, Last appended: 1.166, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:28.251178 3064 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch)
I20240503 07:59:28.251830 3730 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 07:59:28.255537 2665 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch)
I20240503 07:59:28.256407 3730 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 07:59:28.268673 2802 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch)
I20240503 07:59:28.269874 3847 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 167. (index mismatch)
I20240503 07:59:28.269959 2927 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 167. (index mismatch)
I20240503 07:59:28.272522 3730 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s
I20240503 07:59:28.275417 3696 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEADER]: Committing config change with OpId 1.167: config changed from index 126 to 167, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.278218 2258 catalog_manager.cc:5484] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index 126 to 167, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" committed_config { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.279867 2671 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 126 to 167, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.285548 3058 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 126 to 167, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.294149 2797 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 126 to 167, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.329798 3730 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 07:59:28.329973 4467 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 164, Time since last communication: 0.000s
I20240503 07:59:28.332100 2927 raft_consensus.cc:1203] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.165->[1.166-1.168] Dedup: 1.166->[1.167-1.168]
I20240503 07:59:28.332608 2927 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 126 to 167, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.335301 3847 raft_consensus.cc:1203] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.165->[1.166-1.168] Dedup: 1.166->[1.167-1.168]
I20240503 07:59:28.335819 3847 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 126 to 167, 6338e74c39eb4583842e387e64fce522 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.417399 2409 consensus_queue.cc:237] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 1.170, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } }
I20240503 07:59:28.420909 2801 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 07:59:28.421121 3064 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 07:59:28.421402 3392 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 07:59:28.421963 3392 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 07:59:28.441495 3729 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEADER]: Committing config change with OpId 1.171: config changed from index 167 to 171, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.443325 2251 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d116d60ce3a04fc782762c4b8808a178 with cas_config_opid_index 167: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:28.444790 2797 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 167 to 171, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.445324 2258 catalog_manager.cc:5484] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index 167 to 171, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.446861 3064 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 167 to 171, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.448915 2650 tablet_service.cc:1514] Processing DeleteTablet for tablet d116d60ce3a04fc782762c4b8808a178 with delete_type TABLET_DATA_TOMBSTONED (TS 886e4eb306064b98ab923623b8f1d401 not found in new config with opid_index 171) from {username='slave'} at 127.0.0.1:52688
I20240503 07:59:28.453699 4589 tablet_replica.cc:330] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: stopping tablet replica
I20240503 07:59:28.453846 4589 raft_consensus.cc:2229] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.453953 4589 pending_rounds.cc:62] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: Trying to abort 1 pending ops.
I20240503 07:59:28.453987 4589 pending_rounds.cc:68] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: Aborting op as it isn't in flight: id { term: 1 index: 170 } timestamp: 7023506097802199040 op_type: WRITE_OP write_request { tablet_id: "d116d60ce3a04fc782762c4b8808a178" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\243\363\334~\271<&h\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""4b0d15d19026e5ab" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023506097484091392 authz_token { token_data: "\010\204\265\322\261\006\"3\n\005slave\022*\n a67e4da0a0be4cb2b0ca1abeb334fd39\020\001\030\001 \001(\001" signature: "<redacted>""\200\264\270>%GbuR6\313\312:u\344\333\207\345\200|&\326\314\275\360\341\n\210\337\247\034D;\304M\362\tU\226\031\006I\257\344\217>\267\023\363\013l\272,\340\006\326A\336\2730-\345\037w" signing_key_seq_num: 0 } } request_id { client_id: "93457ca4cd664565a2bb96397574b111" seq_no: 426 first_incomplete_seq_no: 426 attempt_no: 0 }
I20240503 07:59:28.454242 4589 raft_consensus.cc:2258] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.455420 2931 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 07:59:28.463575 3847 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240503 07:59:28.469524 4589 ts_tablet_manager.cc:1875] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:28.472718 4589 ts_tablet_manager.cc:1888] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.170
I20240503 07:59:28.472800 4589 log.cc:1196] T d116d60ce3a04fc782762c4b8808a178 P 886e4eb306064b98ab923623b8f1d401: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d116d60ce3a04fc782762c4b8808a178
I20240503 07:59:28.473327 3729 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 07:59:28.473477 2253 catalog_manager.cc:4830] TS 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343): tablet d116d60ce3a04fc782762c4b8808a178 (table rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]) successfully deleted
I20240503 07:59:28.491225 3729 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I20240503 07:59:28.491895 2932 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 167 to 171, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.522392 3847 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 167 to 171, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } }
I20240503 07:59:28.816183 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 07:59:28.822327 2998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.133:46695, user_credentials={real_user=slave}} blocked reactor thread for 47769.6us
I20240503 07:59:28.891500 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:28.895495 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:28.899140 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:28.900177 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:28.953888 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:28.987447 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:29.341678 2671 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 147, Committed index: 147, Last appended: 1.148, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: true } }
I20240503 07:59:29.343801 3848 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 1 index: 149. (index mismatch)
I20240503 07:59:29.344203 4565 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 147, Time since last communication: 0.000s
I20240503 07:59:29.346659 2931 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 149. (index mismatch)
I20240503 07:59:29.347111 4565 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 145, Time since last communication: 0.000s
I20240503 07:59:29.348093 3058 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 1 index: 149. (index mismatch)
I20240503 07:59:29.349298 2539 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 1 index: 149. (index mismatch)
I20240503 07:59:29.350237 3972 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 147, Time since last communication: 0.000s
I20240503 07:59:29.352295 3987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 147, Time since last communication: 0.000s
W20240503 07:59:29.353427 2604 consensus_peers.cc:485] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 -> Peer cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251): Couldn't send request to peer cd82dbc60d0f40688fbe8e712dc7107a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a4acef14f154d0b8fb94428a45a380b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:29.362179 3972 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: true } } }
I20240503 07:59:29.366787 2258 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 reported cstate change: config changed from index 66 to 149, NON_VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:29.365690 2929 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: true } } }
I20240503 07:59:29.381582 3847 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: true } } }
I20240503 07:59:29.414295 3058 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: true } } }
I20240503 07:59:29.417076 2537 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 66 to 149, NON_VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: true } } }
W20240503 07:59:29.516688 2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.131:44343, user_credentials={real_user=slave}} blocked reactor thread for 79700.6us
I20240503 07:59:29.835712 4669 ts_tablet_manager.cc:902] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: Initiating tablet copy from peer 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:29.837282 4669 tablet_copy_client.cc:308] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44343
I20240503 07:59:29.838002 2691 tablet_copy_service.cc:140] P 886e4eb306064b98ab923623b8f1d401: Received BeginTabletCopySession request for tablet 0a4acef14f154d0b8fb94428a45a380b from peer cd82dbc60d0f40688fbe8e712dc7107a ({username='slave'} at 127.0.99.129:38039)
I20240503 07:59:29.838122 2691 tablet_copy_service.cc:161] P 886e4eb306064b98ab923623b8f1d401: Beginning new tablet copy session on tablet 0a4acef14f154d0b8fb94428a45a380b from peer cd82dbc60d0f40688fbe8e712dc7107a at {username='slave'} at 127.0.99.129:38039: session id = cd82dbc60d0f40688fbe8e712dc7107a-0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:29.839073 2691 tablet_copy_source_session.cc:205] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:29.844952 4669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4acef14f154d0b8fb94428a45a380b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.912842 4669 tablet_copy_client.cc:783] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:29.913214 4669 tablet_copy_client.cc:647] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:29.950012 4669 tablet_copy_client.cc:523] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:29.951900 4669 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap starting.
I20240503 07:59:30.351809 4669 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap replayed 1/1 log segments. Stats: ops{read=160 overwritten=0 applied=160 ignored=0} inserts{seen=788 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:30.354588 4669 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: Bootstrap complete.
I20240503 07:59:30.354995 4669 ts_tablet_manager.cc:1367] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: Time spent bootstrapping tablet: real 0.403s user 0.118s sys 0.013s
I20240503 07:59:30.355506 4669 raft_consensus.cc:348] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: true } }
I20240503 07:59:30.355733 4669 raft_consensus.cc:729] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd82dbc60d0f40688fbe8e712dc7107a, State: Initialized, Role: LEARNER
I20240503 07:59:30.355880 4669 consensus_queue.cc:260] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 1.160, Last appended by leader: 160, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: true } }
I20240503 07:59:30.356820 4669 ts_tablet_manager.cc:1398] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:30.357317 2691 tablet_copy_service.cc:342] P 886e4eb306064b98ab923623b8f1d401: Request end of tablet copy session cd82dbc60d0f40688fbe8e712dc7107a-0a4acef14f154d0b8fb94428a45a380b received from {username='slave'} at 127.0.99.129:38039
I20240503 07:59:30.357391 2691 tablet_copy_service.cc:433] P 886e4eb306064b98ab923623b8f1d401: ending tablet copy session cd82dbc60d0f40688fbe8e712dc7107a-0a4acef14f154d0b8fb94428a45a380b on tablet 0a4acef14f154d0b8fb94428a45a380b with peer cd82dbc60d0f40688fbe8e712dc7107a
W20240503 07:59:30.431185 2473 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.129:38251, user_credentials={real_user=slave}} blocked reactor thread for 84120.3us
I20240503 07:59:30.651736 2405 raft_consensus.cc:1203] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.148->[1.149-1.173] Dedup: 1.160->[1.161-1.173]
I20240503 07:59:30.727726 3987 raft_consensus.cc:1050] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: attempting to promote NON_VOTER cd82dbc60d0f40688fbe8e712dc7107a to VOTER
I20240503 07:59:30.728093 3987 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 1.174, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } }
I20240503 07:59:30.730922 2929 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240503 07:59:30.731472 4429 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240503 07:59:30.732561 2407 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEARNER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240503 07:59:30.733786 4429 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240503 07:59:30.737680 2537 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 173. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240503 07:59:30.737709 3847 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240503 07:59:30.738592 4429 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240503 07:59:30.741325 4565 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.175: config changed from index 149 to 175, cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.741950 3063 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240503 07:59:30.744331 3987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 173, Time since last communication: 0.000s
I20240503 07:59:30.744514 2929 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 149 to 175, cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.745149 2539 raft_consensus.cc:1203] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.173->[1.174-1.175] Dedup: 1.174->[1.175-1.175]
I20240503 07:59:30.745585 2539 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 149 to 175, cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.745721 2407 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 149 to 175, cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.747440 2261 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 reported cstate change: config changed from index 149 to 175, cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.748024 3847 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 149 to 175, cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.751564 3987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240503 07:59:30.757095 3060 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 149 to 175, cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.858014 2665 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 178, Committed index: 178, Last appended: 1.178, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } }
I20240503 07:59:30.867532 3845 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240503 07:59:30.868083 3060 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 177. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240503 07:59:30.870627 3987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
I20240503 07:59:30.877548 4670 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 177, Time since last communication: 0.000s
I20240503 07:59:30.880882 2539 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240503 07:59:30.882050 3972 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
I20240503 07:59:30.885998 4565 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.179: config changed from index 175 to 179, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.886466 3845 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 175 to 179, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.889137 3059 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 175 to 179, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.892239 2253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a4acef14f154d0b8fb94428a45a380b with cas_config_opid_index 175: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:30.894515 2409 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240503 07:59:30.894531 2267 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 175 to 179, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.895032 3987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
I20240503 07:59:30.898182 2404 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 175 to 179, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.899302 2911 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a4acef14f154d0b8fb94428a45a380b with delete_type TABLET_DATA_TOMBSTONED (TS e3b61927599645d4b11278833b2f27d6 not found in new config with opid_index 179) from {username='slave'} at 127.0.0.1:36192
I20240503 07:59:30.903816 2539 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 175 to 179, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } }
I20240503 07:59:30.914515 4726 tablet_replica.cc:330] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6: stopping tablet replica
I20240503 07:59:30.915555 4726 raft_consensus.cc:2229] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.915740 4726 raft_consensus.cc:2258] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.936718 4726 ts_tablet_manager.cc:1875] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:30.945457 4726 ts_tablet_manager.cc:1888] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.178
I20240503 07:59:30.950119 4726 log.cc:1196] T 0a4acef14f154d0b8fb94428a45a380b P e3b61927599645d4b11278833b2f27d6: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:30.968811 2252 catalog_manager.cc:4830] TS e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695): tablet 0a4acef14f154d0b8fb94428a45a380b (table rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]) successfully deleted
I20240503 07:59:31.553454 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:31.556591 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:31.594614 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:31.613480 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:31.674013 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:31.743788 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:31.819275 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 07:59:32.103101 2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.134:36145, user_credentials={real_user=slave}} blocked reactor thread for 46599.2us
W20240503 07:59:32.255977 2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.131:44343, user_credentials={real_user=slave}} blocked reactor thread for 57301.5us
I20240503 07:59:32.421914 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:32.424959 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:32.454488 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:32.464607 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:32.466576 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:32.475414 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 07:59:32.501130 2474 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.133:46695, user_credentials={real_user=slave}} blocked reactor thread for 56385.7us
W20240503 07:59:32.564069 2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.134:36145, user_credentials={real_user=slave}} blocked reactor thread for 45908.3us
I20240503 07:59:32.566876 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:33.769793 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:33.792716 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:33.827615 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:33.838330 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:33.851738 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:33.860692 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:33.877290 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 07:59:34.413137 2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.134:36145, user_credentials={real_user=slave}} blocked reactor thread for 47887.2us
I20240503 07:59:34.485683 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:34.514819 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:34.519248 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:34.519860 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:34.539803 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:34.647001 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:34.658720 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:34.975296 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:34.975312 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:34.975981 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:34.977977 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:34.987789 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:34.993206 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:34.995083 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
1a6fc308ffbf49c2b3e717d2a724d973 | 127.0.99.190:42701 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:38429 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0bc088d2657a49b39cb7981f0e3eec21 | 127.0.99.132:37199 | HEALTHY | <none> | 2 | 0
3631cd1b9b79487581e9d31178f671c3 | 127.0.99.130:40355 | HEALTHY | <none> | 6 | 0
6338e74c39eb4583842e387e64fce522 | 127.0.99.135:38745 | HEALTHY | <none> | 0 | 0
81b9495629bb458892b031db80bc3715 | 127.0.99.134:36145 | HEALTHY | <none> | 3 | 0
886e4eb306064b98ab923623b8f1d401 | 127.0.99.131:44343 | HEALTHY | <none> | 3 | 0
cd82dbc60d0f40688fbe8e712dc7107a | 127.0.99.129:38251 | HEALTHY | <none> | 1 | 0
e3b61927599645d4b11278833b2f27d6 | 127.0.99.133:46695 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:38251
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:40355
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:44343
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:37199
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:46695
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:36145
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:38745
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:38251
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:40355
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:44343
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:37199
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:46695
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:36145
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:38745
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:38429 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 11
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 07:59:35.038625 398 log_verifier.cc:126] Checking tablet 02d4dc031bac4eb1a8fa5d25ca7140b3
I20240503 07:59:35.171375 398 log_verifier.cc:177] Verified matching terms for 206 ops in tablet 02d4dc031bac4eb1a8fa5d25ca7140b3
I20240503 07:59:35.171551 398 log_verifier.cc:126] Checking tablet 0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:35.308768 398 log_verifier.cc:177] Verified matching terms for 231 ops in tablet 0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:35.308938 398 log_verifier.cc:126] Checking tablet 401e87d2d48e4515b725c10757178536
I20240503 07:59:35.447074 398 log_verifier.cc:177] Verified matching terms for 255 ops in tablet 401e87d2d48e4515b725c10757178536
I20240503 07:59:35.447237 398 log_verifier.cc:126] Checking tablet 66e126a09c874c51a4c83ccc54ebc2ff
I20240503 07:59:35.576122 398 log_verifier.cc:177] Verified matching terms for 268 ops in tablet 66e126a09c874c51a4c83ccc54ebc2ff
I20240503 07:59:35.576296 398 log_verifier.cc:126] Checking tablet 6a11003820e14dc7b2615f20baf179cc
I20240503 07:59:35.696419 398 log_verifier.cc:177] Verified matching terms for 238 ops in tablet 6a11003820e14dc7b2615f20baf179cc
I20240503 07:59:35.696596 398 log_verifier.cc:126] Checking tablet 71eeb5740a7344749b534273ba1c9922
I20240503 07:59:35.836450 398 log_verifier.cc:177] Verified matching terms for 307 ops in tablet 71eeb5740a7344749b534273ba1c9922
I20240503 07:59:35.836661 398 log_verifier.cc:126] Checking tablet 7ad0b76f04404591b8e4947bbc6f9d73
I20240503 07:59:35.874006 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 7ad0b76f04404591b8e4947bbc6f9d73
I20240503 07:59:35.874068 398 log_verifier.cc:126] Checking tablet 7d732123aab240c4b4216f451e87de09
I20240503 07:59:35.995884 398 log_verifier.cc:177] Verified matching terms for 237 ops in tablet 7d732123aab240c4b4216f451e87de09
I20240503 07:59:35.996048 398 log_verifier.cc:126] Checking tablet 813f904dbfab4d46964bbbf877715cc9
I20240503 07:59:36.116720 398 log_verifier.cc:177] Verified matching terms for 236 ops in tablet 813f904dbfab4d46964bbbf877715cc9
I20240503 07:59:36.116894 398 log_verifier.cc:126] Checking tablet c9d779a081cd48628e24d1321f7b42af
I20240503 07:59:36.235759 398 log_verifier.cc:177] Verified matching terms for 244 ops in tablet c9d779a081cd48628e24d1321f7b42af
I20240503 07:59:36.235914 398 log_verifier.cc:126] Checking tablet ccac60c83bd54650a50a07d87e3fc72a
I20240503 07:59:36.365609 398 log_verifier.cc:177] Verified matching terms for 274 ops in tablet ccac60c83bd54650a50a07d87e3fc72a
I20240503 07:59:36.365789 398 log_verifier.cc:126] Checking tablet d116d60ce3a04fc782762c4b8808a178
I20240503 07:59:36.514611 398 log_verifier.cc:177] Verified matching terms for 286 ops in tablet d116d60ce3a04fc782762c4b8808a178
I20240503 07:59:36.514802 398 log_verifier.cc:126] Checking tablet dbbc403671d24146b48f8e0c14db7de6
I20240503 07:59:36.639155 398 log_verifier.cc:177] Verified matching terms for 233 ops in tablet dbbc403671d24146b48f8e0c14db7de6
I20240503 07:59:36.639325 398 log_verifier.cc:126] Checking tablet dc9685543090435199e2c63872396b0f
I20240503 07:59:36.764472 398 log_verifier.cc:177] Verified matching terms for 259 ops in tablet dc9685543090435199e2c63872396b0f
I20240503 07:59:36.764676 398 log_verifier.cc:126] Checking tablet df62f8897f4f4e4abe13f3455a665031
I20240503 07:59:36.888159 398 log_verifier.cc:177] Verified matching terms for 246 ops in tablet df62f8897f4f4e4abe13f3455a665031
I20240503 07:59:36.888353 398 log_verifier.cc:126] Checking tablet ebe8af8c7edf45368869b33dc1819e35
I20240503 07:59:37.015067 398 log_verifier.cc:177] Verified matching terms for 256 ops in tablet ebe8af8c7edf45368869b33dc1819e35
I20240503 07:59:37.016901 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--builtin_ntp_servers=127.0.99.148:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:37.119194 4997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:37.119372 4997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:37.119426 4997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:37.122324 4997 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:37.122380 4997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:37.122457 4997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:59:37.125890 4997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42701
--never_fsync=true
--heap_profile_path=/tmp/kudu.4997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:37.126600 4997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:37.127559 4997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 07:59:37.135998 4997 server_base.cc:1034] running on GCE node
W20240503 07:59:37.136574 5005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:37.137120 5003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:37.138032 5002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:37.138331 4997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:37.138793 4997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:37.139933 4997 hybrid_clock.cc:648] HybridClock initialized: now 1714723177139917 us; error 25 us; skew 500 ppm
I20240503 07:59:37.146102 4997 webserver.cc:460] Webserver started at http://127.0.99.136:40031/ using document root <none> and password file <none>
I20240503 07:59:37.146553 4997 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:37.146596 4997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:37.146732 4997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:37.148113 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "729d1a41f1cf4537928f250945275517"
format_stamp: "Formatted at 2024-05-03 07:59:37 on dist-test-slave-7b05"
I20240503 07:59:37.148499 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "729d1a41f1cf4537928f250945275517"
format_stamp: "Formatted at 2024-05-03 07:59:37 on dist-test-slave-7b05"
I20240503 07:59:37.151468 4997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:37.153299 5011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:37.154210 4997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:37.154300 4997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "729d1a41f1cf4537928f250945275517"
format_stamp: "Formatted at 2024-05-03 07:59:37 on dist-test-slave-7b05"
I20240503 07:59:37.154369 4997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:37.173440 4997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:37.173951 4997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:37.174074 4997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:37.174517 4997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:37.175267 4997 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:37.175308 4997 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:37.175346 4997 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:37.175374 4997 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:37.183243 4997 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41401
I20240503 07:59:37.183276 5124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41401 every 8 connection(s)
I20240503 07:59:37.184010 4997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:59:37.191660 5125 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42701
I20240503 07:59:37.191846 5125 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:37.192054 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 4997
I20240503 07:59:37.192169 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 07:59:37.192374 5125 heartbeater.cc:511] Master 127.0.99.190:42701 requested a full tablet report, sending...
I20240503 07:59:37.193238 2259 ts_manager.cc:194] Registered new tserver with Master: 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401)
I20240503 07:59:37.193787 2259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:54617
I20240503 07:59:37.373911 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:37.374388 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:37.383965 5059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:37.387959 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:37.390390 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:37.406419 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:37.407986 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:37.408337 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:37.496783 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:37.498696 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:37.502938 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:37.504927 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:37.513199 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:37.529673 5059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:37.531756 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:37.535136 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:37.571018 2797 consensus_queue.cc:237] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.256, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:37.572645 3058 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch)
I20240503 07:59:37.572662 2666 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch)
I20240503 07:59:37.573201 3848 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch)
I20240503 07:59:37.573565 2537 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch)
I20240503 07:59:37.573976 4842 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20240503 07:59:37.574172 5221 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20240503 07:59:37.574321 5222 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20240503 07:59:37.574713 5220 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I20240503 07:59:37.576026 5222 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Committing config change with OpId 1.257: config changed from index 65 to 257, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.576407 3848 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 65 to 257, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
W20240503 07:59:37.577680 2736 consensus_peers.cc:485] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebe8af8c7edf45368869b33dc1819e35. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:37.578583 2666 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 65 to 257, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.578794 2267 catalog_manager.cc:5484] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: config changed from index 65 to 257, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.578918 3058 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 65 to 257, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.579695 2537 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 65 to 257, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.588017 2927 consensus_queue.cc:237] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 1.236, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:37.589566 2666 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20240503 07:59:37.589617 2797 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20240503 07:59:37.589927 3058 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20240503 07:59:37.589994 5240 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240503 07:59:37.590344 3848 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20240503 07:59:37.590821 5239 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240503 07:59:37.591367 5240 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240503 07:59:37.591542 5241 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
W20240503 07:59:37.593464 2867 consensus_peers.cc:485] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 813f904dbfab4d46964bbbf877715cc9. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:37.593482 5241 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 1 LEADER]: Committing config change with OpId 1.237: config changed from index 93 to 237, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.593850 3848 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 93 to 237, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.594836 3058 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 93 to 237, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.594821 2666 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 93 to 237, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.596230 2259 catalog_manager.cc:5484] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 reported cstate change: config changed from index 93 to 237, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e3b61927599645d4b11278833b2f27d6" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.596930 2797 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 93 to 237, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.605139 2666 consensus_queue.cc:237] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:37.606827 2537 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240503 07:59:37.606973 3058 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240503 07:59:37.607161 4995 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240503 07:59:37.607290 2409 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240503 07:59:37.607698 2797 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240503 07:59:37.607959 4995 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240503 07:59:37.608107 5253 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240503 07:59:37.609758 5253 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.610018 5255 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240503 07:59:37.610109 3058 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
W20240503 07:59:37.610248 2605 consensus_peers.cc:485] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02d4dc031bac4eb1a8fa5d25ca7140b3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:37.611202 2797 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.611785 2537 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.612250 2267 catalog_manager.cc:5484] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 reported cstate change: config changed from index -1 to 207, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.612524 2409 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index -1 to 207, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.620985 2666 consensus_queue.cc:237] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 1.268, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:37.622236 3058 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
W20240503 07:59:37.622392 2605 consensus_peers.cc:485] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66e126a09c874c51a4c83ccc54ebc2ff. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:37.622431 2537 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240503 07:59:37.622431 2409 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240503 07:59:37.622784 4995 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240503 07:59:37.622833 2797 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240503 07:59:37.622982 5254 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240503 07:59:37.623167 5255 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240503 07:59:37.623304 5267 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240503 07:59:37.624931 5255 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.269: config changed from index -1 to 269, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.625435 2537 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index -1 to 269, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.625718 2409 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index -1 to 269, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.627072 2797 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index -1 to 269, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.627760 2259 catalog_manager.cc:5484] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 reported cstate change: config changed from index -1 to 269, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.628928 3058 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index -1 to 269, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.637084 3058 consensus_queue.cc:237] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:37.638824 2666 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.638841 2797 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.638828 3848 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.639329 4994 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:37.639595 2927 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.639688 4994 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:37.639961 4994 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:37.640097 5279 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:37.641899 5279 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.642585 2797 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.642813 2666 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.644829 2265 catalog_manager.cc:5484] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: config changed from index -1 to 2, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
W20240503 07:59:37.644953 2998 consensus_peers.cc:485] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ad0b76f04404591b8e4947bbc6f9d73. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:37.645416 3848 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:37.646940 2927 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:38.007586 5338 ts_tablet_manager.cc:902] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:38.008352 5338 tablet_copy_client.cc:308] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44343
I20240503 07:59:38.012018 2691 tablet_copy_service.cc:140] P 886e4eb306064b98ab923623b8f1d401: Received BeginTabletCopySession request for tablet 66e126a09c874c51a4c83ccc54ebc2ff from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:34259)
I20240503 07:59:38.012116 2691 tablet_copy_service.cc:161] P 886e4eb306064b98ab923623b8f1d401: Beginning new tablet copy session on tablet 66e126a09c874c51a4c83ccc54ebc2ff from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:34259: session id = 729d1a41f1cf4537928f250945275517-66e126a09c874c51a4c83ccc54ebc2ff
I20240503 07:59:38.012873 2691 tablet_copy_source_session.cc:205] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:38.016081 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e126a09c874c51a4c83ccc54ebc2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.021514 5338 tablet_copy_client.cc:783] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:38.021682 5338 tablet_copy_client.cc:647] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:38.023237 5338 tablet_copy_client.cc:523] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:38.024910 5338 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:38.044507 5338 log.cc:824] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:38.097914 5345 ts_tablet_manager.cc:902] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:38.098418 5345 tablet_copy_client.cc:308] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:36145
I20240503 07:59:38.103332 3084 tablet_copy_service.cc:140] P 81b9495629bb458892b031db80bc3715: Received BeginTabletCopySession request for tablet 7ad0b76f04404591b8e4947bbc6f9d73 from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:56269)
I20240503 07:59:38.103449 3084 tablet_copy_service.cc:161] P 81b9495629bb458892b031db80bc3715: Beginning new tablet copy session on tablet 7ad0b76f04404591b8e4947bbc6f9d73 from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:56269: session id = 729d1a41f1cf4537928f250945275517-7ad0b76f04404591b8e4947bbc6f9d73
I20240503 07:59:38.104418 3084 tablet_copy_source_session.cc:205] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:38.105304 5345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad0b76f04404591b8e4947bbc6f9d73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.107355 5345 tablet_copy_client.cc:783] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:38.107486 5345 tablet_copy_client.cc:647] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:38.108464 5345 tablet_copy_client.cc:523] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:38.109567 5345 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:38.113693 5345 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:38.113963 5345 tablet_bootstrap.cc:492] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:38.114284 5345 ts_tablet_manager.cc:1367] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 07:59:38.116847 5345 raft_consensus.cc:348] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.117136 5345 raft_consensus.cc:729] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:38.117488 5345 consensus_queue.cc:260] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.118129 5125 heartbeater.cc:503] Master 127.0.99.190:42701 was elected leader, sending a full tablet report...
I20240503 07:59:38.118378 5345 ts_tablet_manager.cc:1398] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:38.118908 3084 tablet_copy_service.cc:342] P 81b9495629bb458892b031db80bc3715: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-7ad0b76f04404591b8e4947bbc6f9d73 received from {username='slave'} at 127.0.99.136:56269
I20240503 07:59:38.118970 3084 tablet_copy_service.cc:433] P 81b9495629bb458892b031db80bc3715: ending tablet copy session 729d1a41f1cf4537928f250945275517-7ad0b76f04404591b8e4947bbc6f9d73 on tablet 7ad0b76f04404591b8e4947bbc6f9d73 with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:38.182170 5345 ts_tablet_manager.cc:902] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199)
I20240503 07:59:38.182615 5345 tablet_copy_client.cc:308] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37199
I20240503 07:59:38.187570 2820 tablet_copy_service.cc:140] P 0bc088d2657a49b39cb7981f0e3eec21: Received BeginTabletCopySession request for tablet ebe8af8c7edf45368869b33dc1819e35 from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:44499)
I20240503 07:59:38.187664 2820 tablet_copy_service.cc:161] P 0bc088d2657a49b39cb7981f0e3eec21: Beginning new tablet copy session on tablet ebe8af8c7edf45368869b33dc1819e35 from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:44499: session id = 729d1a41f1cf4537928f250945275517-ebe8af8c7edf45368869b33dc1819e35
I20240503 07:59:38.188372 2820 tablet_copy_source_session.cc:205] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:38.189256 5345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8af8c7edf45368869b33dc1819e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.191255 5345 tablet_copy_client.cc:783] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:38.191401 5345 tablet_copy_client.cc:647] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:38.192235 5382 ts_tablet_manager.cc:902] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695)
I20240503 07:59:38.192881 5382 tablet_copy_client.cc:308] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:46695
I20240503 07:59:38.193521 5345 tablet_copy_client.cc:523] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:38.193725 5338 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=269 overwritten=0 applied=269 ignored=0} inserts{seen=900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:38.193986 5338 tablet_bootstrap.cc:492] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:38.194182 5338 ts_tablet_manager.cc:1367] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.169s user 0.160s sys 0.008s
I20240503 07:59:38.194646 5338 raft_consensus.cc:348] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.194761 5338 raft_consensus.cc:729] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:38.194856 5338 consensus_queue.cc:260] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 269, Last appended: 1.269, Last appended by leader: 269, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.195556 5345 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:38.196777 2953 tablet_copy_service.cc:140] P e3b61927599645d4b11278833b2f27d6: Received BeginTabletCopySession request for tablet 813f904dbfab4d46964bbbf877715cc9 from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:37611)
I20240503 07:59:38.196887 2953 tablet_copy_service.cc:161] P e3b61927599645d4b11278833b2f27d6: Beginning new tablet copy session on tablet 813f904dbfab4d46964bbbf877715cc9 from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:37611: session id = 729d1a41f1cf4537928f250945275517-813f904dbfab4d46964bbbf877715cc9
I20240503 07:59:38.197844 2953 tablet_copy_source_session.cc:205] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:38.195194 5338 ts_tablet_manager.cc:1398] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:38.198688 5382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813f904dbfab4d46964bbbf877715cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.199849 2691 tablet_copy_service.cc:342] P 886e4eb306064b98ab923623b8f1d401: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-66e126a09c874c51a4c83ccc54ebc2ff received from {username='slave'} at 127.0.99.136:34259
I20240503 07:59:38.199934 2691 tablet_copy_service.cc:433] P 886e4eb306064b98ab923623b8f1d401: ending tablet copy session 729d1a41f1cf4537928f250945275517-66e126a09c874c51a4c83ccc54ebc2ff on tablet 66e126a09c874c51a4c83ccc54ebc2ff with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:38.200609 5382 tablet_copy_client.cc:783] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:38.200738 5382 tablet_copy_client.cc:647] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:38.202824 5382 tablet_copy_client.cc:523] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:38.203702 5382 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:38.213380 5338 ts_tablet_manager.cc:902] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:38.213728 5338 tablet_copy_client.cc:308] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44343
I20240503 07:59:38.214109 2691 tablet_copy_service.cc:140] P 886e4eb306064b98ab923623b8f1d401: Received BeginTabletCopySession request for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:34259)
I20240503 07:59:38.214228 2691 tablet_copy_service.cc:161] P 886e4eb306064b98ab923623b8f1d401: Beginning new tablet copy session on tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:34259: session id = 729d1a41f1cf4537928f250945275517-02d4dc031bac4eb1a8fa5d25ca7140b3
I20240503 07:59:38.215122 2691 tablet_copy_source_session.cc:205] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:38.215757 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.217555 5338 tablet_copy_client.cc:783] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:38.217690 5338 tablet_copy_client.cc:647] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:38.222877 5338 tablet_copy_client.cc:523] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:38.223819 5338 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:38.377041 5345 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=257 overwritten=0 applied=257 ignored=0} inserts{seen=939 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:38.377360 5345 tablet_bootstrap.cc:492] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:38.377549 5345 ts_tablet_manager.cc:1367] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.182s user 0.155s sys 0.020s
I20240503 07:59:38.377990 5345 raft_consensus.cc:348] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.378126 5345 raft_consensus.cc:729] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:38.378223 5345 consensus_queue.cc:260] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 257, Last appended: 1.257, Last appended by leader: 257, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.378484 5345 ts_tablet_manager.cc:1398] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.378878 2820 tablet_copy_service.cc:342] P 0bc088d2657a49b39cb7981f0e3eec21: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-ebe8af8c7edf45368869b33dc1819e35 received from {username='slave'} at 127.0.99.136:44499
I20240503 07:59:38.378963 2820 tablet_copy_service.cc:433] P 0bc088d2657a49b39cb7981f0e3eec21: ending tablet copy session 729d1a41f1cf4537928f250945275517-ebe8af8c7edf45368869b33dc1819e35 on tablet ebe8af8c7edf45368869b33dc1819e35 with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:38.386682 5382 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=237 overwritten=0 applied=237 ignored=0} inserts{seen=919 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:38.386965 5382 tablet_bootstrap.cc:492] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:38.387146 5382 ts_tablet_manager.cc:1367] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.183s user 0.164s sys 0.008s
I20240503 07:59:38.387526 5382 raft_consensus.cc:348] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.387645 5382 raft_consensus.cc:729] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:38.387739 5382 consensus_queue.cc:260] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 237, Last appended: 1.237, Last appended by leader: 237, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.388013 5382 ts_tablet_manager.cc:1398] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.388479 5338 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=207 overwritten=0 applied=207 ignored=0} inserts{seen=892 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:38.388494 2953 tablet_copy_service.cc:342] P e3b61927599645d4b11278833b2f27d6: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-813f904dbfab4d46964bbbf877715cc9 received from {username='slave'} at 127.0.99.136:37611
I20240503 07:59:38.388552 2953 tablet_copy_service.cc:433] P e3b61927599645d4b11278833b2f27d6: ending tablet copy session 729d1a41f1cf4537928f250945275517-813f904dbfab4d46964bbbf877715cc9 on tablet 813f904dbfab4d46964bbbf877715cc9 with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:38.388689 5338 tablet_bootstrap.cc:492] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:38.388849 5338 ts_tablet_manager.cc:1367] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.165s user 0.144s sys 0.014s
I20240503 07:59:38.389282 5338 raft_consensus.cc:348] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.389390 5338 raft_consensus.cc:729] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:38.389485 5338 consensus_queue.cc:260] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 207, Last appended: 1.207, Last appended by leader: 207, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:38.389709 5338 ts_tablet_manager.cc:1398] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:38.390137 2691 tablet_copy_service.cc:342] P 886e4eb306064b98ab923623b8f1d401: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-02d4dc031bac4eb1a8fa5d25ca7140b3 received from {username='slave'} at 127.0.99.136:34259
I20240503 07:59:38.390205 2691 tablet_copy_service.cc:433] P 886e4eb306064b98ab923623b8f1d401: ending tablet copy session 729d1a41f1cf4537928f250945275517-02d4dc031bac4eb1a8fa5d25ca7140b3 on tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:38.395562 5079 raft_consensus.cc:1203] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.268->[1.269-1.269] Dedup: 1.269->[]
I20240503 07:59:38.686523 5079 raft_consensus.cc:1203] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:38.731782 5079 raft_consensus.cc:1203] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.206->[1.207-1.207] Dedup: 1.207->[]
I20240503 07:59:38.736001 5079 raft_consensus.cc:1203] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.236->[1.237-1.237] Dedup: 1.237->[]
I20240503 07:59:38.803195 5079 raft_consensus.cc:1203] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.256->[1.257-1.257] Dedup: 1.257->[]
I20240503 07:59:38.959401 5267 raft_consensus.cc:1050] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:38.959869 5267 consensus_queue.cc:237] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 1.269, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:38.961405 3058 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240503 07:59:38.961405 2409 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240503 07:59:38.961599 2797 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240503 07:59:38.961652 5079 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240503 07:59:38.961848 5479 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240503 07:59:38.962083 2537 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240503 07:59:38.962092 5267 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240503 07:59:38.962281 5479 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240503 07:59:38.963049 5477 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240503 07:59:38.963240 5267 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240503 07:59:38.965121 5479 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.270: config changed from index 269 to 270, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.965888 2409 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 269 to 270, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.966141 2537 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 269 to 270, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.966374 5079 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 269 to 270, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.969115 2265 catalog_manager.cc:5484] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index 269 to 270, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.970117 3058 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 269 to 270, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.970119 2797 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 269 to 270, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } attrs { replace: true } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.975705 2666 consensus_queue.cc:237] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 1.270, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:38.976842 5079 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch)
I20240503 07:59:38.976855 2409 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch)
I20240503 07:59:38.976930 2797 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch)
I20240503 07:59:38.977144 3058 raft_consensus.cc:1261] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch)
I20240503 07:59:38.977246 5479 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20240503 07:59:38.977408 5478 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20240503 07:59:38.977572 5255 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20240503 07:59:38.977748 5254 consensus_queue.cc:1035] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20240503 07:59:38.979166 5254 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.271: config changed from index 270 to 271, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.979794 5079 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 270 to 271, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.979919 2797 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 270 to 271, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.980273 3058 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 270 to 271, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.980806 2253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66e126a09c874c51a4c83ccc54ebc2ff with cas_config_opid_index 270: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:38.981447 2409 raft_consensus.cc:2932] T 66e126a09c874c51a4c83ccc54ebc2ff P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 270 to 271, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:38.982292 2259 catalog_manager.cc:5484] T 66e126a09c874c51a4c83ccc54ebc2ff P 886e4eb306064b98ab923623b8f1d401 reported cstate change: config changed from index 270 to 271, VOTER 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:38.985253 2518 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e126a09c874c51a4c83ccc54ebc2ff with delete_type TABLET_DATA_TOMBSTONED (TS 3631cd1b9b79487581e9d31178f671c3 not found in new config with opid_index 271) from {username='slave'} at 127.0.0.1:52904
I20240503 07:59:38.985836 5489 tablet_replica.cc:330] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: stopping tablet replica
I20240503 07:59:38.985953 5489 raft_consensus.cc:2229] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.986085 5489 raft_consensus.cc:2258] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.987128 5489 ts_tablet_manager.cc:1875] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:38.988757 5489 ts_tablet_manager.cc:1888] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.270
I20240503 07:59:38.988816 5489 log.cc:1196] T 66e126a09c874c51a4c83ccc54ebc2ff P 3631cd1b9b79487581e9d31178f671c3: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/66e126a09c874c51a4c83ccc54ebc2ff
I20240503 07:59:38.989240 2251 catalog_manager.cc:4830] TS 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355): tablet 66e126a09c874c51a4c83ccc54ebc2ff (table rebalance_test_table_0 [id=47bd34c090d04940ba0ab84776e9fabb]) successfully deleted
I20240503 07:59:39.043637 2409 consensus_queue.cc:237] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 1.286, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:39.045545 2797 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 287. (index mismatch)
I20240503 07:59:39.046025 3848 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 287. (index mismatch)
I20240503 07:59:39.046119 3058 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 287. (index mismatch)
I20240503 07:59:39.046135 2927 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 287. (index mismatch)
I20240503 07:59:39.046448 5522 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20240503 07:59:39.046689 5523 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20240503 07:59:39.046852 5524 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20240503 07:59:39.047008 3729 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20240503 07:59:39.049203 5524 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEADER]: Committing config change with OpId 1.287: config changed from index 171 to 287, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
W20240503 07:59:39.049371 2343 consensus_peers.cc:485] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d116d60ce3a04fc782762c4b8808a178. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.049703 3058 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.287: config changed from index 171 to 287, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.049875 2797 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.287: config changed from index 171 to 287, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.052613 2266 catalog_manager.cc:5484] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: config changed from index 171 to 287, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.052740 3848 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.287: config changed from index 171 to 287, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.054546 2927 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.287: config changed from index 171 to 287, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.099915 5445 raft_consensus.cc:1050] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:39.100327 5445 consensus_queue.cc:237] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:39.101781 3845 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:39.101828 5079 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:39.101908 2666 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:39.102289 2927 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:39.102705 5434 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:39.102845 2797 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:39.102886 5559 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:39.103062 5558 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:39.103184 5445 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:39.103312 5281 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:39.105722 5445 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.106127 3845 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.106408 2666 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.107868 5079 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.108177 2797 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.109283 2927 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.109357 2266 catalog_manager.cc:5484] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 2 to 3, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:39.112385 3059 consensus_queue.cc:237] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:39.113678 3845 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:39.113710 5079 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:39.114005 5281 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:39.114171 5558 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:39.114800 2666 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:39.114836 2797 raft_consensus.cc:1261] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:39.115110 5558 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:39.115250 5281 consensus_queue.cc:1035] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:39.116578 5281 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.117290 5079 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.117959 2253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7ad0b76f04404591b8e4947bbc6f9d73 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:39.118454 2666 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.119025 2267 catalog_manager.cc:5484] T 7ad0b76f04404591b8e4947bbc6f9d73 P 729d1a41f1cf4537928f250945275517 reported cstate change: config changed from index 3 to 4, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.120206 2797 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.120321 3845 raft_consensus.cc:2932] T 7ad0b76f04404591b8e4947bbc6f9d73 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3b61927599645d4b11278833b2f27d6 (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.121641 2911 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ad0b76f04404591b8e4947bbc6f9d73 with delete_type TABLET_DATA_TOMBSTONED (TS e3b61927599645d4b11278833b2f27d6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36192
I20240503 07:59:39.123966 5565 tablet_replica.cc:330] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: stopping tablet replica
I20240503 07:59:39.124099 5565 raft_consensus.cc:2229] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.124253 5565 raft_consensus.cc:2258] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.125268 5565 ts_tablet_manager.cc:1875] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:39.126909 5565 ts_tablet_manager.cc:1888] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:39.126967 5565 log.cc:1196] T 7ad0b76f04404591b8e4947bbc6f9d73 P e3b61927599645d4b11278833b2f27d6: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7ad0b76f04404591b8e4947bbc6f9d73
I20240503 07:59:39.127393 2252 catalog_manager.cc:4830] TS e3b61927599645d4b11278833b2f27d6 (127.0.99.133:46695): tablet 7ad0b76f04404591b8e4947bbc6f9d73 (table TestTable [id=9fda7ceb7e054b4c906909f1f94103be]) successfully deleted
I20240503 07:59:39.287470 5447 raft_consensus.cc:1050] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:39.287941 5447 consensus_queue.cc:237] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.237, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:39.289353 5079 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 07:59:39.290115 3059 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 07:59:39.290151 2666 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 07:59:39.290211 3845 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 07:59:39.290522 2797 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 07:59:39.290558 5576 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 07:59:39.290952 5447 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 07:59:39.291111 5581 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 07:59:39.291409 5580 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 07:59:39.292086 5579 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 07:59:39.294497 5447 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 1 LEADER]: Committing config change with OpId 1.238: config changed from index 237 to 238, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.295965 2666 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 237 to 238, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.296360 5079 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 237 to 238, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.296597 3058 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 237 to 238, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.298629 3845 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 237 to 238, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.298674 2267 catalog_manager.cc:5484] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517 reported cstate change: config changed from index 237 to 238, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3b61927599645d4b11278833b2f27d6" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.301263 5479 raft_consensus.cc:1050] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:39.301688 5479 consensus_queue.cc:237] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 1.207, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:39.303515 2797 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 07:59:39.305137 2409 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 07:59:39.305500 2537 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 07:59:39.305835 5255 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 07:59:39.306011 5478 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 07:59:39.308261 5478 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 07:59:39.309877 5079 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 07:59:39.310249 5255 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 07:59:39.311345 3058 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20240503 07:59:39.312139 5254 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20240503 07:59:39.312090 5479 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.208: config changed from index 207 to 208, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.313164 3058 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.314858 2927 consensus_queue.cc:237] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 1.238, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:39.314817 2537 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.314944 2261 catalog_manager.cc:5484] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 reported cstate change: config changed from index 207 to 208, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:39.316227 3059 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240503 07:59:39.316527 2800 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.316886 5079 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240503 07:59:39.316951 2801 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240503 07:59:39.317668 2797 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 237 to 238, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.317997 5447 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 07:59:39.318028 3845 raft_consensus.cc:1261] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer e3b61927599645d4b11278833b2f27d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240503 07:59:39.318173 5079 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.318303 5447 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240503 07:59:39.318869 5578 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240503 07:59:39.319550 5447 consensus_queue.cc:1035] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240503 07:59:39.321668 5447 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P e3b61927599645d4b11278833b2f27d6 [term 1 LEADER]: Committing config change with OpId 1.239: config changed from index 238 to 239, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.322515 3845 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.323189 3059 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.323287 2409 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.325780 2258 catalog_manager.cc:5484] T 813f904dbfab4d46964bbbf877715cc9 P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 238 to 239, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e3b61927599645d4b11278833b2f27d6" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.326814 2797 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.327533 5078 raft_consensus.cc:2932] T 813f904dbfab4d46964bbbf877715cc9 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.328292 2252 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 813f904dbfab4d46964bbbf877715cc9 with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:39.329995 2650 tablet_service.cc:1514] Processing DeleteTablet for tablet 813f904dbfab4d46964bbbf877715cc9 with delete_type TABLET_DATA_TOMBSTONED (TS 886e4eb306064b98ab923623b8f1d401 not found in new config with opid_index 239) from {username='slave'} at 127.0.0.1:52688
I20240503 07:59:39.330356 5603 tablet_replica.cc:330] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401: stopping tablet replica
I20240503 07:59:39.330514 5603 raft_consensus.cc:2229] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.330695 5603 raft_consensus.cc:2258] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.331831 5603 ts_tablet_manager.cc:1875] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:39.333750 5603 ts_tablet_manager.cc:1888] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.238
I20240503 07:59:39.333809 5603 log.cc:1196] T 813f904dbfab4d46964bbbf877715cc9 P 886e4eb306064b98ab923623b8f1d401: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/813f904dbfab4d46964bbbf877715cc9
I20240503 07:59:39.334266 2253 catalog_manager.cc:4830] TS 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343): tablet 813f904dbfab4d46964bbbf877715cc9 (table rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]) successfully deleted
I20240503 07:59:39.341949 2537 consensus_queue.cc:237] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.237, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:39.344014 2666 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 07:59:39.344023 2927 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 07:59:39.344386 5475 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 07:59:39.344612 2409 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 07:59:39.345038 3845 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240503 07:59:39.345564 5617 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 07:59:39.345746 5475 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 07:59:39.346081 5475 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
W20240503 07:59:39.347733 2474 consensus_peers.cc:485] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d732123aab240c4b4216f451e87de09. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.347994 5617 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.238: config changed from index 67 to 238, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.348364 2409 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 67 to 238, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.348886 2666 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 67 to 238, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.349269 3845 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 67 to 238, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.351151 2261 catalog_manager.cc:5484] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index 67 to 238, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.352562 2927 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 67 to 238, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.360247 3059 consensus_queue.cc:237] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 1.244, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:39.361721 2666 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240503 07:59:39.361729 2797 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240503 07:59:39.361725 2409 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240503 07:59:39.362063 5558 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20240503 07:59:39.362236 5434 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20240503 07:59:39.362318 2927 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 245. (index mismatch)
I20240503 07:59:39.362366 5281 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
W20240503 07:59:39.362534 2998 consensus_peers.cc:485] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9d779a081cd48628e24d1321f7b42af. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.362706 5559 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20240503 07:59:39.364322 5559 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.245: config changed from index 71 to 245, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.364974 2409 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 71 to 245, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.365026 2666 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 71 to 245, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.365034 2797 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 71 to 245, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.366322 2927 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.245: config changed from index 71 to 245, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.367318 2259 catalog_manager.cc:5484] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: config changed from index 71 to 245, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.391212 5464 raft_consensus.cc:1050] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:39.391652 5464 consensus_queue.cc:237] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:39.393234 2537 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240503 07:59:39.393249 5078 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240503 07:59:39.393394 3845 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240503 07:59:39.393507 2665 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240503 07:59:39.393687 5662 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240503 07:59:39.393797 3059 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240503 07:59:39.394012 5662 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240503 07:59:39.394248 5464 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240503 07:59:39.394430 5662 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240503 07:59:39.395138 5653 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240503 07:59:39.396903 5653 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.397796 3845 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.397837 2665 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.399489 2537 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.399544 2259 catalog_manager.cc:5484] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: config changed from index 257 to 258, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:39.400573 5078 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.401157 3059 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.406278 2797 consensus_queue.cc:237] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 1.258, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:39.407371 2537 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20240503 07:59:39.407410 5078 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20240503 07:59:39.407430 3845 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20240503 07:59:39.407420 2665 raft_consensus.cc:1261] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 0bc088d2657a49b39cb7981f0e3eec21: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20240503 07:59:39.407743 5668 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240503 07:59:39.407913 5464 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240503 07:59:39.408062 5653 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240503 07:59:39.408210 5662 consensus_queue.cc:1035] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240503 07:59:39.410017 5464 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 LEADER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.410346 2665 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.410670 2537 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.410719 3845 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.411798 2254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ebe8af8c7edf45368869b33dc1819e35 with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:39.412364 2261 catalog_manager.cc:5484] T ebe8af8c7edf45368869b33dc1819e35 P 0bc088d2657a49b39cb7981f0e3eec21 reported cstate change: config changed from index 258 to 259, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "0bc088d2657a49b39cb7981f0e3eec21" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:39.412681 5078 raft_consensus.cc:2932] T ebe8af8c7edf45368869b33dc1819e35 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.415100 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe8af8c7edf45368869b33dc1819e35 with delete_type TABLET_DATA_TOMBSTONED (TS 81b9495629bb458892b031db80bc3715 not found in new config with opid_index 259) from {username='slave'} at 127.0.0.1:37648
I20240503 07:59:39.416165 5669 tablet_replica.cc:330] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715: stopping tablet replica
I20240503 07:59:39.416378 5669 raft_consensus.cc:2229] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.416543 5669 raft_consensus.cc:2258] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.417811 5669 ts_tablet_manager.cc:1875] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:39.419342 5669 ts_tablet_manager.cc:1888] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.258
I20240503 07:59:39.419399 5669 log.cc:1196] T ebe8af8c7edf45368869b33dc1819e35 P 81b9495629bb458892b031db80bc3715: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ebe8af8c7edf45368869b33dc1819e35
I20240503 07:59:39.419782 2253 catalog_manager.cc:4830] TS 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145): tablet ebe8af8c7edf45368869b33dc1819e35 (table rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]) successfully deleted
I20240503 07:59:39.465415 5670 ts_tablet_manager.cc:902] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251)
I20240503 07:59:39.465827 5670 tablet_copy_client.cc:308] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38251
I20240503 07:59:39.469094 2429 tablet_copy_service.cc:140] P cd82dbc60d0f40688fbe8e712dc7107a: Received BeginTabletCopySession request for tablet d116d60ce3a04fc782762c4b8808a178 from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:59311)
I20240503 07:59:39.469190 2429 tablet_copy_service.cc:161] P cd82dbc60d0f40688fbe8e712dc7107a: Beginning new tablet copy session on tablet d116d60ce3a04fc782762c4b8808a178 from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:59311: session id = 729d1a41f1cf4537928f250945275517-d116d60ce3a04fc782762c4b8808a178
I20240503 07:59:39.469874 2429 tablet_copy_source_session.cc:205] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:39.470592 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116d60ce3a04fc782762c4b8808a178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.472143 5670 tablet_copy_client.cc:783] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:39.472275 5670 tablet_copy_client.cc:647] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:39.473690 5670 tablet_copy_client.cc:523] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:39.474668 5670 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:39.636307 2665 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.231, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:39.638026 2409 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20240503 07:59:39.638090 3845 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20240503 07:59:39.638563 3059 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20240503 07:59:39.638938 5477 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20240503 07:59:39.639225 2537 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20240503 07:59:39.639261 5267 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20240503 07:59:39.639729 5267 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
W20240503 07:59:39.640724 2605 consensus_peers.cc:485] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a4acef14f154d0b8fb94428a45a380b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.641290 5267 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20240503 07:59:39.641911 5477 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.232: config changed from index 179 to 232, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.643714 3845 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 179 to 232, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.644469 3059 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 179 to 232, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.645504 2258 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 reported cstate change: config changed from index 179 to 232, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:39.647948 2409 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 179 to 232, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.647997 2537 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 179 to 232, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.656145 5670 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=287 overwritten=0 applied=287 ignored=0} inserts{seen=956 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:39.656456 5670 tablet_bootstrap.cc:492] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:39.656659 5670 ts_tablet_manager.cc:1367] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.182s user 0.163s sys 0.016s
I20240503 07:59:39.657034 5670 raft_consensus.cc:348] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:39.657169 5670 raft_consensus.cc:729] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:39.657272 5670 consensus_queue.cc:260] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 287, Last appended: 1.287, Last appended by leader: 287, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:39.657549 5670 ts_tablet_manager.cc:1398] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:39.657963 2429 tablet_copy_service.cc:342] P cd82dbc60d0f40688fbe8e712dc7107a: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-d116d60ce3a04fc782762c4b8808a178 received from {username='slave'} at 127.0.99.136:59311
I20240503 07:59:39.658046 2429 tablet_copy_service.cc:433] P cd82dbc60d0f40688fbe8e712dc7107a: ending tablet copy session 729d1a41f1cf4537928f250945275517-d116d60ce3a04fc782762c4b8808a178 on tablet d116d60ce3a04fc782762c4b8808a178 with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:39.738385 5670 ts_tablet_manager.cc:902] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355)
I20240503 07:59:39.738821 5670 tablet_copy_client.cc:308] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40355
I20240503 07:59:39.741925 2560 tablet_copy_service.cc:140] P 3631cd1b9b79487581e9d31178f671c3: Received BeginTabletCopySession request for tablet 7d732123aab240c4b4216f451e87de09 from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:48193)
I20240503 07:59:39.742017 2560 tablet_copy_service.cc:161] P 3631cd1b9b79487581e9d31178f671c3: Beginning new tablet copy session on tablet 7d732123aab240c4b4216f451e87de09 from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:48193: session id = 729d1a41f1cf4537928f250945275517-7d732123aab240c4b4216f451e87de09
I20240503 07:59:39.742826 2560 tablet_copy_source_session.cc:205] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:39.743610 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d732123aab240c4b4216f451e87de09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.745229 5670 tablet_copy_client.cc:783] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:39.745362 5670 tablet_copy_client.cc:647] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:39.746613 5670 tablet_copy_client.cc:523] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:39.747555 5670 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:39.844316 2665 consensus_queue.cc:237] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 1.208, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:39.845687 5078 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20240503 07:59:39.845731 2797 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20240503 07:59:39.845755 3059 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20240503 07:59:39.846356 5478 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20240503 07:59:39.846688 5478 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20240503 07:59:39.847151 2537 raft_consensus.cc:1261] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 1 index: 209. (index mismatch)
I20240503 07:59:39.848378 5254 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20240503 07:59:39.848990 5267 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.850083 3059 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.851083 5078 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.851469 2253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 with cas_config_opid_index 208: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:39.852865 5477 consensus_queue.cc:1035] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20240503 07:59:39.853845 2797 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.854056 2266 catalog_manager.cc:5484] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 208 to 209, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.856860 2389 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 with delete_type TABLET_DATA_TOMBSTONED (TS cd82dbc60d0f40688fbe8e712dc7107a not found in new config with opid_index 209) from {username='slave'} at 127.0.0.1:59282
I20240503 07:59:39.857995 5742 tablet_replica.cc:330] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a: stopping tablet replica
I20240503 07:59:39.858124 5742 raft_consensus.cc:2229] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.858280 5742 raft_consensus.cc:2258] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.859012 2537 raft_consensus.cc:2932] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.209: config changed from index 208 to 209, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:39.860270 5742 ts_tablet_manager.cc:1875] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:39.862440 5742 ts_tablet_manager.cc:1888] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.208
I20240503 07:59:39.862505 5742 log.cc:1196] T 02d4dc031bac4eb1a8fa5d25ca7140b3 P cd82dbc60d0f40688fbe8e712dc7107a: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/02d4dc031bac4eb1a8fa5d25ca7140b3
I20240503 07:59:39.863178 2251 catalog_manager.cc:4830] TS cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251): tablet 02d4dc031bac4eb1a8fa5d25ca7140b3 (table rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]) successfully deleted
I20240503 07:59:39.867331 5743 ts_tablet_manager.cc:902] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145)
I20240503 07:59:39.867889 5743 tablet_copy_client.cc:308] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:36145
I20240503 07:59:39.868289 3084 tablet_copy_service.cc:140] P 81b9495629bb458892b031db80bc3715: Received BeginTabletCopySession request for tablet c9d779a081cd48628e24d1321f7b42af from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:56269)
I20240503 07:59:39.868379 3084 tablet_copy_service.cc:161] P 81b9495629bb458892b031db80bc3715: Beginning new tablet copy session on tablet c9d779a081cd48628e24d1321f7b42af from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:56269: session id = 729d1a41f1cf4537928f250945275517-c9d779a081cd48628e24d1321f7b42af
I20240503 07:59:39.869278 3084 tablet_copy_source_session.cc:205] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:39.870002 5743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d779a081cd48628e24d1321f7b42af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.871454 5743 tablet_copy_client.cc:783] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:39.871583 5743 tablet_copy_client.cc:647] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:39.873435 5743 tablet_copy_client.cc:523] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:39.873641 5078 raft_consensus.cc:1203] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.286->[1.287-1.287] Dedup: 1.287->[]
I20240503 07:59:39.874502 5743 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:39.889161 5670 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=238 overwritten=0 applied=238 ignored=0} inserts{seen=866 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:39.889428 5670 tablet_bootstrap.cc:492] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:39.889616 5670 ts_tablet_manager.cc:1367] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.142s user 0.134s sys 0.008s
I20240503 07:59:39.890045 5670 raft_consensus.cc:348] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:39.890156 5670 raft_consensus.cc:729] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:39.890262 5670 consensus_queue.cc:260] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 238, Last appended: 1.238, Last appended by leader: 238, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:39.890570 5670 ts_tablet_manager.cc:1398] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:39.891153 2560 tablet_copy_service.cc:342] P 3631cd1b9b79487581e9d31178f671c3: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-7d732123aab240c4b4216f451e87de09 received from {username='slave'} at 127.0.99.136:48193
I20240503 07:59:39.891216 2560 tablet_copy_service.cc:433] P 3631cd1b9b79487581e9d31178f671c3: ending tablet copy session 729d1a41f1cf4537928f250945275517-7d732123aab240c4b4216f451e87de09 on tablet 7d732123aab240c4b4216f451e87de09 with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:39.919441 2537 consensus_queue.cc:237] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 1.233, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:39.921301 3059 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 07:59:39.921588 2927 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 07:59:39.921588 3845 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 07:59:39.921710 2409 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
W20240503 07:59:39.922005 2474 consensus_peers.cc:485] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 -> Peer 729d1a41f1cf4537928f250945275517 (127.0.99.136:41401): Couldn't send request to peer 729d1a41f1cf4537928f250945275517. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbbc403671d24146b48f8e0c14db7de6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.922005 5617 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240503 07:59:39.922222 4415 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240503 07:59:39.922472 4415 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240503 07:59:39.922695 5782 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.001s
I20240503 07:59:39.924407 5617 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.234: config changed from index 108 to 234, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.925313 3059 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 108 to 234, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.927157 2267 catalog_manager.cc:5484] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index 108 to 234, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:39.928021 2927 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 108 to 234, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:39.936867 2409 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 108 to 234, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:40.022518 5670 ts_tablet_manager.cc:902] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343)
I20240503 07:59:40.022953 5670 tablet_copy_client.cc:308] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44343
I20240503 07:59:40.023424 2691 tablet_copy_service.cc:140] P 886e4eb306064b98ab923623b8f1d401: Received BeginTabletCopySession request for tablet 0a4acef14f154d0b8fb94428a45a380b from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:34259)
I20240503 07:59:40.023526 2691 tablet_copy_service.cc:161] P 886e4eb306064b98ab923623b8f1d401: Beginning new tablet copy session on tablet 0a4acef14f154d0b8fb94428a45a380b from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:34259: session id = 729d1a41f1cf4537928f250945275517-0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:40.024508 2691 tablet_copy_source_session.cc:205] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:40.025449 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4acef14f154d0b8fb94428a45a380b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.027549 5670 tablet_copy_client.cc:783] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:40.027694 5670 tablet_copy_client.cc:647] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:40.030480 5670 tablet_copy_client.cc:523] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:40.031761 5670 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:40.036307 5743 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=245 overwritten=0 applied=245 ignored=0} inserts{seen=873 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:40.036602 5743 tablet_bootstrap.cc:492] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:40.036792 5743 ts_tablet_manager.cc:1367] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.162s user 0.145s sys 0.013s
I20240503 07:59:40.037266 5743 raft_consensus.cc:348] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:40.037418 5743 raft_consensus.cc:729] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:40.037533 5743 consensus_queue.cc:260] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 245, Last appended: 1.245, Last appended by leader: 245, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 245 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:40.037854 5743 ts_tablet_manager.cc:1398] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:40.038437 3084 tablet_copy_service.cc:342] P 81b9495629bb458892b031db80bc3715: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-c9d779a081cd48628e24d1321f7b42af received from {username='slave'} at 127.0.99.136:56269
I20240503 07:59:40.038519 3084 tablet_copy_service.cc:433] P 81b9495629bb458892b031db80bc3715: ending tablet copy session 729d1a41f1cf4537928f250945275517-c9d779a081cd48628e24d1321f7b42af on tablet c9d779a081cd48628e24d1321f7b42af with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:40.173882 5670 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=232 overwritten=0 applied=232 ignored=0} inserts{seen=857 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:40.174168 5670 tablet_bootstrap.cc:492] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:40.174377 5670 ts_tablet_manager.cc:1367] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.143s user 0.120s sys 0.012s
I20240503 07:59:40.174856 5670 raft_consensus.cc:348] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:40.174978 5670 raft_consensus.cc:729] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:40.175071 5670 consensus_queue.cc:260] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 232, Last appended: 1.232, Last appended by leader: 232, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:40.175346 5670 ts_tablet_manager.cc:1398] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:40.175843 2691 tablet_copy_service.cc:342] P 886e4eb306064b98ab923623b8f1d401: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-0a4acef14f154d0b8fb94428a45a380b received from {username='slave'} at 127.0.99.136:34259
I20240503 07:59:40.175917 2691 tablet_copy_service.cc:433] P 886e4eb306064b98ab923623b8f1d401: ending tablet copy session 729d1a41f1cf4537928f250945275517-0a4acef14f154d0b8fb94428a45a380b on tablet 0a4acef14f154d0b8fb94428a45a380b with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:40.190263 5078 raft_consensus.cc:1203] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.237->[1.238-1.238] Dedup: 1.238->[]
I20240503 07:59:40.256593 5749 raft_consensus.cc:1050] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:40.257019 5749 consensus_queue.cc:237] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 1.287, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:40.258620 2797 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 07:59:40.258620 3845 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 07:59:40.258695 5078 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 07:59:40.258764 3059 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 07:59:40.259088 5846 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20240503 07:59:40.259307 2927 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 07:59:40.259377 5748 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20240503 07:59:40.259641 5847 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20240503 07:59:40.259856 5749 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20240503 07:59:40.260010 3729 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20240503 07:59:40.261992 5846 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEADER]: Committing config change with OpId 1.288: config changed from index 287 to 288, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.262585 5078 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.262956 3059 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.264304 2797 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.265033 2259 catalog_manager.cc:5484] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517 reported cstate change: config changed from index 287 to 288, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.267385 3845 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.268926 2927 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 287 to 288, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.271816 2409 consensus_queue.cc:237] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 1.288, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:40.272934 5078 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch)
I20240503 07:59:40.273103 3845 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch)
I20240503 07:59:40.273146 2927 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch)
I20240503 07:59:40.273537 5846 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20240503 07:59:40.273655 5748 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20240503 07:59:40.273875 3059 raft_consensus.cc:1261] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer cd82dbc60d0f40688fbe8e712dc7107a: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch)
I20240503 07:59:40.273941 5748 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20240503 07:59:40.274576 5748 consensus_queue.cc:1035] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I20240503 07:59:40.275535 5846 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 LEADER]: Committing config change with OpId 1.289: config changed from index 288 to 289, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.276137 5078 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 288 to 289, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.277129 3059 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 288 to 289, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.278388 2251 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d116d60ce3a04fc782762c4b8808a178 with cas_config_opid_index 288: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:40.278782 2266 catalog_manager.cc:5484] T d116d60ce3a04fc782762c4b8808a178 P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index 288 to 289, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:40.279156 3845 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 288 to 289, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.279634 2927 raft_consensus.cc:2932] T d116d60ce3a04fc782762c4b8808a178 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 288 to 289, VOTER 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132) evicted. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.281211 2781 tablet_service.cc:1514] Processing DeleteTablet for tablet d116d60ce3a04fc782762c4b8808a178 with delete_type TABLET_DATA_TOMBSTONED (TS 0bc088d2657a49b39cb7981f0e3eec21 not found in new config with opid_index 289) from {username='slave'} at 127.0.0.1:39912
I20240503 07:59:40.281548 5854 tablet_replica.cc:330] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21: stopping tablet replica
I20240503 07:59:40.281746 5854 raft_consensus.cc:2229] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.281920 5854 raft_consensus.cc:2258] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.283322 5854 ts_tablet_manager.cc:1875] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:40.285391 5854 ts_tablet_manager.cc:1888] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.288
I20240503 07:59:40.285459 5854 log.cc:1196] T d116d60ce3a04fc782762c4b8808a178 P 0bc088d2657a49b39cb7981f0e3eec21: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d116d60ce3a04fc782762c4b8808a178
I20240503 07:59:40.285951 2254 catalog_manager.cc:4830] TS 0bc088d2657a49b39cb7981f0e3eec21 (127.0.99.132:37199): tablet d116d60ce3a04fc782762c4b8808a178 (table rebalance_test_table_4 [id=a67e4da0a0be4cb2b0ca1abeb334fd39]) successfully deleted
I20240503 07:59:40.338384 5670 ts_tablet_manager.cc:902] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: Initiating tablet copy from peer 3631cd1b9b79487581e9d31178f671c3 (127.0.99.130:40355)
I20240503 07:59:40.338811 5670 tablet_copy_client.cc:308] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:40355
I20240503 07:59:40.339221 2560 tablet_copy_service.cc:140] P 3631cd1b9b79487581e9d31178f671c3: Received BeginTabletCopySession request for tablet dbbc403671d24146b48f8e0c14db7de6 from peer 729d1a41f1cf4537928f250945275517 ({username='slave'} at 127.0.99.136:48193)
I20240503 07:59:40.339315 2560 tablet_copy_service.cc:161] P 3631cd1b9b79487581e9d31178f671c3: Beginning new tablet copy session on tablet dbbc403671d24146b48f8e0c14db7de6 from peer 729d1a41f1cf4537928f250945275517 at {username='slave'} at 127.0.99.136:48193: session id = 729d1a41f1cf4537928f250945275517-dbbc403671d24146b48f8e0c14db7de6
I20240503 07:59:40.340068 2560 tablet_copy_source_session.cc:205] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:40.340817 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbc403671d24146b48f8e0c14db7de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:40.342478 5670 tablet_copy_client.cc:783] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:40.342617 5670 tablet_copy_client.cc:647] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:40.343780 5670 tablet_copy_client.cc:523] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:40.344676 5670 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: Bootstrap starting.
I20240503 07:59:40.374677 5078 raft_consensus.cc:1203] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.244->[1.245-1.245] Dedup: 1.245->[]
I20240503 07:59:40.381726 3845 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 108 to 234, NON_VOTER 729d1a41f1cf4537928f250945275517 (127.0.99.136) added. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } } }
I20240503 07:59:40.488147 5670 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: Bootstrap replayed 1/1 log segments. Stats: ops{read=234 overwritten=0 applied=234 ignored=0} inserts{seen=916 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:40.488425 5670 tablet_bootstrap.cc:492] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: Bootstrap complete.
I20240503 07:59:40.488605 5670 ts_tablet_manager.cc:1367] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: Time spent bootstrapping tablet: real 0.144s user 0.131s sys 0.009s
I20240503 07:59:40.488989 5670 raft_consensus.cc:348] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:40.489137 5670 raft_consensus.cc:729] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 729d1a41f1cf4537928f250945275517, State: Initialized, Role: LEARNER
I20240503 07:59:40.489231 5670 consensus_queue.cc:260] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 234, Last appended: 1.234, Last appended by leader: 234, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: true } }
I20240503 07:59:40.489514 5670 ts_tablet_manager.cc:1398] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:40.490012 2560 tablet_copy_service.cc:342] P 3631cd1b9b79487581e9d31178f671c3: Request end of tablet copy session 729d1a41f1cf4537928f250945275517-dbbc403671d24146b48f8e0c14db7de6 received from {username='slave'} at 127.0.99.136:48193
I20240503 07:59:40.490095 2560 tablet_copy_service.cc:433] P 3631cd1b9b79487581e9d31178f671c3: ending tablet copy session 729d1a41f1cf4537928f250945275517-dbbc403671d24146b48f8e0c14db7de6 on tablet dbbc403671d24146b48f8e0c14db7de6 with peer 729d1a41f1cf4537928f250945275517
I20240503 07:59:40.522181 5078 raft_consensus.cc:1203] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.231->[1.232-1.232] Dedup: 1.232->[]
I20240503 07:59:40.767282 5078 raft_consensus.cc:1203] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.233->[1.234-1.234] Dedup: 1.234->[]
I20240503 07:59:40.801888 5784 raft_consensus.cc:1050] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:40.802335 5784 consensus_queue.cc:237] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 1.238, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:40.804021 5078 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240503 07:59:40.804133 3845 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240503 07:59:40.804533 2665 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240503 07:59:40.804576 2927 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240503 07:59:40.804576 2409 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240503 07:59:40.804955 5931 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240503 07:59:40.805258 5788 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240503 07:59:40.805424 5933 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240503 07:59:40.805568 5932 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240503 07:59:40.805888 5782 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240503 07:59:40.807976 5782 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.239: config changed from index 238 to 239, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.808393 2409 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.809188 2665 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.809535 3845 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.811060 5078 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.811105 2267 catalog_manager.cc:5484] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index 238 to 239, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:40.811357 2927 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 238 to 239, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.815380 2537 consensus_queue.cc:237] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 1.239, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:40.816622 2927 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
I20240503 07:59:40.816643 5078 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
I20240503 07:59:40.817082 2665 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
I20240503 07:59:40.817040 5933 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20240503 07:59:40.817196 2409 raft_consensus.cc:1261] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
I20240503 07:59:40.817234 5932 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20240503 07:59:40.817409 5788 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20240503 07:59:40.817531 5931 consensus_queue.cc:1035] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s
I20240503 07:59:40.818958 5784 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.240: config changed from index 239 to 240, VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) evicted. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.819474 2665 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 239 to 240, VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) evicted. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.820138 5078 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 239 to 240, VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) evicted. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.820719 2251 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d732123aab240c4b4216f451e87de09 with cas_config_opid_index 239: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:40.821127 2409 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 239 to 240, VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) evicted. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.821319 2927 raft_consensus.cc:2932] T 7d732123aab240c4b4216f451e87de09 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 239 to 240, VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) evicted. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.822095 2259 catalog_manager.cc:5484] T 7d732123aab240c4b4216f451e87de09 P 3631cd1b9b79487581e9d31178f671c3 reported cstate change: config changed from index 239 to 240, VOTER 6338e74c39eb4583842e387e64fce522 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:40.828516 3828 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d732123aab240c4b4216f451e87de09 with delete_type TABLET_DATA_TOMBSTONED (TS 6338e74c39eb4583842e387e64fce522 not found in new config with opid_index 240) from {username='slave'} at 127.0.0.1:45380
I20240503 07:59:40.828960 5944 tablet_replica.cc:330] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: stopping tablet replica
I20240503 07:59:40.829277 5944 raft_consensus.cc:2229] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.829506 5944 raft_consensus.cc:2258] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.830873 5944 ts_tablet_manager.cc:1875] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:40.832639 5944 ts_tablet_manager.cc:1888] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.239
I20240503 07:59:40.832706 5944 log.cc:1196] T 7d732123aab240c4b4216f451e87de09 P 6338e74c39eb4583842e387e64fce522: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/7d732123aab240c4b4216f451e87de09
I20240503 07:59:40.833135 2252 catalog_manager.cc:4830] TS 6338e74c39eb4583842e387e64fce522 (127.0.99.135:38745): tablet 7d732123aab240c4b4216f451e87de09 (table rebalance_test_table_2 [id=b305a3a5cdc84cf28aa9445aa6b2879a]) successfully deleted
I20240503 07:59:40.840616 5861 raft_consensus.cc:1050] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:40.841004 5861 consensus_queue.cc:237] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 1.245, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:40.842439 5078 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20240503 07:59:40.842581 2665 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20240503 07:59:40.842615 2409 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20240503 07:59:40.842711 2927 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20240503 07:59:40.842839 5947 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20240503 07:59:40.842909 2797 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch)
I20240503 07:59:40.843070 5948 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20240503 07:59:40.843288 5861 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20240503 07:59:40.843401 5946 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20240503 07:59:40.843511 5860 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I20240503 07:59:40.845769 5947 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.246: config changed from index 245 to 246, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.846112 2409 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.846395 5078 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.847528 2797 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.848011 2665 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.848163 2927 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 245 to 246, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.849368 2258 catalog_manager.cc:5484] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a reported cstate change: config changed from index 245 to 246, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } attrs { promote: false replace: true } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.853368 3058 consensus_queue.cc:237] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 1.246, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:40.854494 5078 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240503 07:59:40.854791 5947 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20240503 07:59:40.854838 2927 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240503 07:59:40.854980 2409 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240503 07:59:40.855407 2797 raft_consensus.cc:1261] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Refusing update from remote peer 81b9495629bb458892b031db80bc3715: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240503 07:59:40.855250 5947 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20240503 07:59:40.855803 5861 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20240503 07:59:40.856329 5946 consensus_queue.cc:1035] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20240503 07:59:40.857281 5948 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 [term 1 LEADER]: Committing config change with OpId 1.247: config changed from index 246 to 247, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.857758 2927 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 246 to 247, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.857769 2797 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 0bc088d2657a49b39cb7981f0e3eec21 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 246 to 247, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.858945 2253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9d779a081cd48628e24d1321f7b42af with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:40.859253 2409 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 246 to 247, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.859750 2258 catalog_manager.cc:5484] T c9d779a081cd48628e24d1321f7b42af P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 246 to 247, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "81b9495629bb458892b031db80bc3715" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:40.861477 5078 raft_consensus.cc:2932] T c9d779a081cd48628e24d1321f7b42af P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 246 to 247, VOTER 886e4eb306064b98ab923623b8f1d401 (127.0.99.131) evicted. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "0bc088d2657a49b39cb7981f0e3eec21" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37199 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:40.861769 2650 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d779a081cd48628e24d1321f7b42af with delete_type TABLET_DATA_TOMBSTONED (TS 886e4eb306064b98ab923623b8f1d401 not found in new config with opid_index 247) from {username='slave'} at 127.0.0.1:52688
I20240503 07:59:40.862079 5954 tablet_replica.cc:330] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: stopping tablet replica
I20240503 07:59:40.862259 5954 raft_consensus.cc:2229] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:40.862452 5954 raft_consensus.cc:2258] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:40.864229 5954 ts_tablet_manager.cc:1875] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:40.865778 5954 ts_tablet_manager.cc:1888] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.246
I20240503 07:59:40.865836 5954 log.cc:1196] T c9d779a081cd48628e24d1321f7b42af P 886e4eb306064b98ab923623b8f1d401: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c9d779a081cd48628e24d1321f7b42af
I20240503 07:59:40.866273 2253 catalog_manager.cc:4830] TS 886e4eb306064b98ab923623b8f1d401 (127.0.99.131:44343): tablet c9d779a081cd48628e24d1321f7b42af (table rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]) successfully deleted
I20240503 07:59:41.012248 5891 raft_consensus.cc:1050] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:41.012660 5891 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 232, Committed index: 232, Last appended: 1.232, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:41.014240 3058 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240503 07:59:41.014256 2537 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240503 07:59:41.014642 5986 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 07:59:41.014703 5078 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240503 07:59:41.014848 5987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 07:59:41.015029 5478 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 07:59:41.016228 3845 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240503 07:59:41.016558 2409 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240503 07:59:41.016896 5987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 07:59:41.017407 5986 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 232, Time since last communication: 0.000s
I20240503 07:59:41.018550 5478 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.233: config changed from index 232 to 233, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.018955 2537 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.019044 3845 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.020430 5078 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.021360 3058 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.021733 2265 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 reported cstate change: config changed from index 232 to 233, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.022750 2409 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index 232 to 233, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } attrs { replace: true } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.026790 2665 consensus_queue.cc:237] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 1.233, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:41.027909 5078 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 07:59:41.028025 2537 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 07:59:41.028025 3845 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 07:59:41.028219 5987 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240503 07:59:41.028412 5986 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240503 07:59:41.028537 2409 raft_consensus.cc:1261] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 886e4eb306064b98ab923623b8f1d401: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch)
I20240503 07:59:41.028581 5891 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240503 07:59:41.028909 5986 consensus_queue.cc:1035] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240503 07:59:41.030376 5986 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 [term 1 LEADER]: Committing config change with OpId 1.234: config changed from index 233 to 234, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.030941 2537 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 3631cd1b9b79487581e9d31178f671c3 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 233 to 234, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.030963 3845 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 233 to 234, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.031872 2253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a4acef14f154d0b8fb94428a45a380b with cas_config_opid_index 233: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:41.032563 5078 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 233 to 234, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.032672 2409 raft_consensus.cc:2932] T 0a4acef14f154d0b8fb94428a45a380b P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 233 to 234, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.033385 2265 catalog_manager.cc:5484] T 0a4acef14f154d0b8fb94428a45a380b P 886e4eb306064b98ab923623b8f1d401 reported cstate change: config changed from index 233 to 234, VOTER 81b9495629bb458892b031db80bc3715 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "886e4eb306064b98ab923623b8f1d401" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "886e4eb306064b98ab923623b8f1d401" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:41.036288 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a4acef14f154d0b8fb94428a45a380b with delete_type TABLET_DATA_TOMBSTONED (TS 81b9495629bb458892b031db80bc3715 not found in new config with opid_index 234) from {username='slave'} at 127.0.0.1:37648
I20240503 07:59:41.037274 5993 tablet_replica.cc:330] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715: stopping tablet replica
I20240503 07:59:41.037386 5993 raft_consensus.cc:2229] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:41.037524 5993 raft_consensus.cc:2258] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:41.038709 5993 ts_tablet_manager.cc:1875] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:41.040526 5993 ts_tablet_manager.cc:1888] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.233
I20240503 07:59:41.040586 5993 log.cc:1196] T 0a4acef14f154d0b8fb94428a45a380b P 81b9495629bb458892b031db80bc3715: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:41.041013 2253 catalog_manager.cc:4830] TS 81b9495629bb458892b031db80bc3715 (127.0.99.134:36145): tablet 0a4acef14f154d0b8fb94428a45a380b (table rebalance_test_table_3 [id=ebbfdc5ee9834ed6908c57b61e31351d]) successfully deleted
I20240503 07:59:41.145781 5933 raft_consensus.cc:1050] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3: attempting to promote NON_VOTER 729d1a41f1cf4537928f250945275517 to VOTER
I20240503 07:59:41.146210 5933 consensus_queue.cc:237] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:41.147668 5078 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517 [term 1 LEARNER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240503 07:59:41.147779 3845 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240503 07:59:41.148131 3058 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240503 07:59:41.148124 2927 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240503 07:59:41.148277 2409 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240503 07:59:41.148283 5933 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 07:59:41.148440 5782 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 07:59:41.148700 5931 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 07:59:41.148859 5932 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 07:59:41.149091 5782 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 07:59:41.151098 5932 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.235: config changed from index 234 to 235, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.151643 5078 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 234 to 235, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.151830 3058 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 234 to 235, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.152360 2409 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 234 to 235, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.154381 2261 catalog_manager.cc:5484] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 234 to 235, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.154897 2927 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 234 to 235, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.154963 3845 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 234 to 235, 729d1a41f1cf4537928f250945275517 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "cd82dbc60d0f40688fbe8e712dc7107a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38251 } attrs { replace: true } } peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.159058 2537 consensus_queue.cc:237] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.235, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } }
I20240503 07:59:41.160290 5078 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20240503 07:59:41.160578 5782 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240503 07:59:41.160856 2927 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20240503 07:59:41.160907 3845 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20240503 07:59:41.161270 3058 raft_consensus.cc:1261] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Refusing update from remote peer 3631cd1b9b79487581e9d31178f671c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20240503 07:59:41.161592 5784 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240503 07:59:41.161741 5782 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240503 07:59:41.161871 5933 consensus_queue.cc:1035] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240503 07:59:41.163614 5933 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 3631cd1b9b79487581e9d31178f671c3 [term 1 LEADER]: Committing config change with OpId 1.236: config changed from index 235 to 236, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.164640 3845 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 6338e74c39eb4583842e387e64fce522 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 235 to 236, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.164674 3058 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 235 to 236, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.166322 2251 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dbbc403671d24146b48f8e0c14db7de6 with cas_config_opid_index 235: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:41.166916 2258 catalog_manager.cc:5484] T dbbc403671d24146b48f8e0c14db7de6 P 81b9495629bb458892b031db80bc3715 reported cstate change: config changed from index 235 to 236, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "3631cd1b9b79487581e9d31178f671c3" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.167430 5078 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P 729d1a41f1cf4537928f250945275517 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 235 to 236, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.169344 2927 raft_consensus.cc:2932] T dbbc403671d24146b48f8e0c14db7de6 P e3b61927599645d4b11278833b2f27d6 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 235 to 236, VOTER cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e3b61927599645d4b11278833b2f27d6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46695 } } peers { permanent_uuid: "3631cd1b9b79487581e9d31178f671c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40355 } } peers { permanent_uuid: "81b9495629bb458892b031db80bc3715" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36145 } } peers { permanent_uuid: "6338e74c39eb4583842e387e64fce522" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "729d1a41f1cf4537928f250945275517" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41401 } attrs { promote: false } } }
I20240503 07:59:41.171043 2389 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbc403671d24146b48f8e0c14db7de6 with delete_type TABLET_DATA_TOMBSTONED (TS cd82dbc60d0f40688fbe8e712dc7107a not found in new config with opid_index 236) from {username='slave'} at 127.0.0.1:59282
I20240503 07:59:41.171370 6014 tablet_replica.cc:330] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a: stopping tablet replica
I20240503 07:59:41.171489 6014 raft_consensus.cc:2229] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:41.171633 6014 raft_consensus.cc:2258] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:41.172845 6014 ts_tablet_manager.cc:1875] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:41.174640 6014 ts_tablet_manager.cc:1888] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.235
I20240503 07:59:41.174714 6014 log.cc:1196] T dbbc403671d24146b48f8e0c14db7de6 P cd82dbc60d0f40688fbe8e712dc7107a: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbbc403671d24146b48f8e0c14db7de6
I20240503 07:59:41.175218 2251 catalog_manager.cc:4830] TS cd82dbc60d0f40688fbe8e712dc7107a (127.0.99.129:38251): tablet dbbc403671d24146b48f8e0c14db7de6 (table rebalance_test_table_1 [id=ca2a48a109274416a6bf71098168e532]) successfully deleted
I20240503 07:59:41.372782 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:41.373320 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.380813 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:41.381222 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:41.389259 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:41.390368 5059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:41.390579 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:41.425689 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:41.497324 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:41.498617 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.498975 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:41.499894 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:41.505936 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:41.527011 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:41.527120 5059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:41.527487 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:41.623070 2518 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:41.623836 2781 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:41.626019 5059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:41.630477 2650 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:41.637230 3828 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:41.637447 2389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:41.646076 2911 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:41.659525 3042 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
1a6fc308ffbf49c2b3e717d2a724d973 | 127.0.99.190:42701 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:38429 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0bc088d2657a49b39cb7981f0e3eec21 | 127.0.99.132:37199 | HEALTHY | <none> | 2 | 0
3631cd1b9b79487581e9d31178f671c3 | 127.0.99.130:40355 | HEALTHY | <none> | 6 | 0
6338e74c39eb4583842e387e64fce522 | 127.0.99.135:38745 | HEALTHY | <none> | 0 | 0
729d1a41f1cf4537928f250945275517 | 127.0.99.136:41401 | HEALTHY | <none> | 0 | 0
81b9495629bb458892b031db80bc3715 | 127.0.99.134:36145 | HEALTHY | <none> | 3 | 0
886e4eb306064b98ab923623b8f1d401 | 127.0.99.131:44343 | HEALTHY | <none> | 3 | 0
cd82dbc60d0f40688fbe8e712dc7107a | 127.0.99.129:38251 | HEALTHY | <none> | 1 | 0
e3b61927599645d4b11278833b2f27d6 | 127.0.99.133:46695 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:38251
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:40355
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:44343
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:37199
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:46695
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:36145
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:38745
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:41401
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:38251
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:40355
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:44343
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:37199
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:46695
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:36145
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:38745
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:41401
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:38429 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 07:59:41.683451 398 log_verifier.cc:126] Checking tablet 02d4dc031bac4eb1a8fa5d25ca7140b3
I20240503 07:59:41.779819 398 log_verifier.cc:177] Verified matching terms for 209 ops in tablet 02d4dc031bac4eb1a8fa5d25ca7140b3
I20240503 07:59:41.779965 398 log_verifier.cc:126] Checking tablet 0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:41.883795 398 log_verifier.cc:177] Verified matching terms for 234 ops in tablet 0a4acef14f154d0b8fb94428a45a380b
I20240503 07:59:41.883955 398 log_verifier.cc:126] Checking tablet 401e87d2d48e4515b725c10757178536
I20240503 07:59:41.991973 398 log_verifier.cc:177] Verified matching terms for 255 ops in tablet 401e87d2d48e4515b725c10757178536
I20240503 07:59:41.992146 398 log_verifier.cc:126] Checking tablet 66e126a09c874c51a4c83ccc54ebc2ff
I20240503 07:59:42.110002 398 log_verifier.cc:177] Verified matching terms for 271 ops in tablet 66e126a09c874c51a4c83ccc54ebc2ff
I20240503 07:59:42.110191 398 log_verifier.cc:126] Checking tablet 6a11003820e14dc7b2615f20baf179cc
I20240503 07:59:42.215019 398 log_verifier.cc:177] Verified matching terms for 238 ops in tablet 6a11003820e14dc7b2615f20baf179cc
I20240503 07:59:42.215180 398 log_verifier.cc:126] Checking tablet 71eeb5740a7344749b534273ba1c9922
I20240503 07:59:42.346364 398 log_verifier.cc:177] Verified matching terms for 307 ops in tablet 71eeb5740a7344749b534273ba1c9922
I20240503 07:59:42.346558 398 log_verifier.cc:126] Checking tablet 7ad0b76f04404591b8e4947bbc6f9d73
I20240503 07:59:42.371613 398 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 7ad0b76f04404591b8e4947bbc6f9d73
I20240503 07:59:42.371671 398 log_verifier.cc:126] Checking tablet 7d732123aab240c4b4216f451e87de09
I20240503 07:59:42.476706 398 log_verifier.cc:177] Verified matching terms for 240 ops in tablet 7d732123aab240c4b4216f451e87de09
I20240503 07:59:42.476874 398 log_verifier.cc:126] Checking tablet 813f904dbfab4d46964bbbf877715cc9
I20240503 07:59:42.597570 398 log_verifier.cc:177] Verified matching terms for 239 ops in tablet 813f904dbfab4d46964bbbf877715cc9
I20240503 07:59:42.597746 398 log_verifier.cc:126] Checking tablet c9d779a081cd48628e24d1321f7b42af
I20240503 07:59:42.711494 398 log_verifier.cc:177] Verified matching terms for 247 ops in tablet c9d779a081cd48628e24d1321f7b42af
I20240503 07:59:42.711681 398 log_verifier.cc:126] Checking tablet ccac60c83bd54650a50a07d87e3fc72a
I20240503 07:59:42.825006 398 log_verifier.cc:177] Verified matching terms for 274 ops in tablet ccac60c83bd54650a50a07d87e3fc72a
I20240503 07:59:42.825201 398 log_verifier.cc:126] Checking tablet d116d60ce3a04fc782762c4b8808a178
I20240503 07:59:42.947273 398 log_verifier.cc:177] Verified matching terms for 289 ops in tablet d116d60ce3a04fc782762c4b8808a178
I20240503 07:59:42.947475 398 log_verifier.cc:126] Checking tablet dbbc403671d24146b48f8e0c14db7de6
I20240503 07:59:43.053148 398 log_verifier.cc:177] Verified matching terms for 236 ops in tablet dbbc403671d24146b48f8e0c14db7de6
I20240503 07:59:43.053337 398 log_verifier.cc:126] Checking tablet dc9685543090435199e2c63872396b0f
I20240503 07:59:43.167742 398 log_verifier.cc:177] Verified matching terms for 259 ops in tablet dc9685543090435199e2c63872396b0f
I20240503 07:59:43.167896 398 log_verifier.cc:126] Checking tablet df62f8897f4f4e4abe13f3455a665031
I20240503 07:59:43.278327 398 log_verifier.cc:177] Verified matching terms for 246 ops in tablet df62f8897f4f4e4abe13f3455a665031
I20240503 07:59:43.278491 398 log_verifier.cc:126] Checking tablet ebe8af8c7edf45368869b33dc1819e35
I20240503 07:59:43.391996 398 log_verifier.cc:177] Verified matching terms for 259 ops in tablet ebe8af8c7edf45368869b33dc1819e35
I20240503 07:59:43.393604 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 2327
I20240503 07:59:43.400465 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 2458
I20240503 07:59:43.406422 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 2589
I20240503 07:59:43.412164 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 2720
I20240503 07:59:43.417461 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 2851
I20240503 07:59:43.422282 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 2982
I20240503 07:59:43.427281 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 3763
I20240503 07:59:43.431846 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 4997
I20240503 07:59:43.436424 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 2236
2024-05-03T07:59:43Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/7 (31526 ms)
[----------] 1 test from RebalanceParamTest (31526 ms total)
[----------] 1 test from TserverAddedDuringRebalancingTest
[ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1
I20240503 07:59:43.515260 398 test_util.cc:268] Using random seed: 440260214
I20240503 07:59:43.516013 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:43.516053 398 ts_itest-base.cc:113] --------------
I20240503 07:59:43.516062 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 07:59:43.516072 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:59:43.516080 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:43Z Disabled control of system clock
I20240503 07:59:43.523818 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42987
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42987
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 07:59:43.620611 6133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:43.620806 6133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:43.620858 6133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:43.623716 6133 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:43.623781 6133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:43.623814 6133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:43.623845 6133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:43.623870 6133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:43.627277 6133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42987
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:43.628002 6133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:43.628957 6133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:43.636083 6138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.636600 6139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:43.637629 6133 server_base.cc:1034] running on GCE node
W20240503 07:59:43.638279 6141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:43.638707 6133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:43.639498 6133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:43.640636 6133 hybrid_clock.cc:648] HybridClock initialized: now 1714723183640626 us; error 25 us; skew 500 ppm
I20240503 07:59:43.646234 6133 webserver.cc:460] Webserver started at http://127.0.99.190:35185/ using document root <none> and password file <none>
I20240503 07:59:43.646718 6133 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:43.646773 6133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:43.646924 6133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:43.648398 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "56c0b66969e4477fbb9ae5d329115122"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-7b05"
I20240503 07:59:43.648813 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "56c0b66969e4477fbb9ae5d329115122"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-7b05"
I20240503 07:59:43.651649 6133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:43.653486 6147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.654353 6133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:43.654462 6133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "56c0b66969e4477fbb9ae5d329115122"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-7b05"
I20240503 07:59:43.654543 6133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:43.669390 6133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:43.669873 6133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:43.670006 6133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:43.675735 6133 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42987
I20240503 07:59:43.675767 6199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42987 every 8 connection(s)
I20240503 07:59:43.676548 6133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:43.677542 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 6133
I20240503 07:59:43.677650 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:43.679298 6200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:43.683791 6200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122: Bootstrap starting.
I20240503 07:59:43.685909 6200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:43.686620 6200 log.cc:824] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:43.688045 6200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122: No bootstrap required, opened a new log
I20240503 07:59:43.690461 6200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c0b66969e4477fbb9ae5d329115122" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42987 } }
I20240503 07:59:43.690624 6200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:43.690680 6200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c0b66969e4477fbb9ae5d329115122, State: Initialized, Role: FOLLOWER
I20240503 07:59:43.691051 6200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c0b66969e4477fbb9ae5d329115122" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42987 } }
I20240503 07:59:43.691177 6200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:43.691275 6200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:43.691392 6200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:43.692015 6200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c0b66969e4477fbb9ae5d329115122" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42987 } }
I20240503 07:59:43.692291 6200 leader_election.cc:302] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56c0b66969e4477fbb9ae5d329115122; no voters:
I20240503 07:59:43.692512 6200 leader_election.cc:288] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:43.692587 6205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:43.692766 6205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [term 1 LEADER]: Becoming Leader. State: Replica: 56c0b66969e4477fbb9ae5d329115122, State: Running, Role: LEADER
I20240503 07:59:43.693032 6205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c0b66969e4477fbb9ae5d329115122" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42987 } }
I20240503 07:59:43.693413 6200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:43.694490 6207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56c0b66969e4477fbb9ae5d329115122. Latest consensus state: current_term: 1 leader_uuid: "56c0b66969e4477fbb9ae5d329115122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c0b66969e4477fbb9ae5d329115122" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42987 } } }
I20240503 07:59:43.694602 6207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:43.695040 6206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56c0b66969e4477fbb9ae5d329115122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c0b66969e4477fbb9ae5d329115122" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42987 } } }
I20240503 07:59:43.695135 6206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:43.695482 6212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:43.697976 6212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:43.702456 6212 catalog_manager.cc:1347] Generated new cluster ID: 98c038e9c82549969ab9776a7ff2c8b4
I20240503 07:59:43.702510 6212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:43.709012 6212 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:43.710045 6212 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:43.718998 6212 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 56c0b66969e4477fbb9ae5d329115122: Generated new TSK 0
I20240503 07:59:43.719628 6212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:43.721920 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42987
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 07:59:43.819553 6224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:43.819754 6224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:43.819816 6224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:43.822755 6224 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:43.822816 6224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:43.822896 6224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:43.826336 6224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.6224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:43.827123 6224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:43.828138 6224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:43.836448 6229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.837339 6230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.838039 6232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:43.838742 6224 server_base.cc:1034] running on GCE node
I20240503 07:59:43.839079 6224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:43.839563 6224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:43.840724 6224 hybrid_clock.cc:648] HybridClock initialized: now 1714723183840697 us; error 40 us; skew 500 ppm
I20240503 07:59:43.842447 6224 webserver.cc:460] Webserver started at http://127.0.99.129:35869/ using document root <none> and password file <none>
I20240503 07:59:43.842957 6224 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:43.843006 6224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:43.843156 6224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:43.844666 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "253dd189c3c542248402cba222394f43"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-7b05"
I20240503 07:59:43.845139 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "253dd189c3c542248402cba222394f43"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-7b05"
I20240503 07:59:43.848093 6224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:43.850045 6238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.850912 6224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:43.850998 6224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "253dd189c3c542248402cba222394f43"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-7b05"
I20240503 07:59:43.851076 6224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:43.862087 6224 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:43.862581 6224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:43.862710 6224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:43.863159 6224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:43.863934 6224 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:43.863977 6224 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.864017 6224 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:43.864043 6224 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:43.871516 6224 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:38721
I20240503 07:59:43.871546 6351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:38721 every 8 connection(s)
I20240503 07:59:43.872292 6224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:43.876796 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 6224
I20240503 07:59:43.876917 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:43.878798 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42987
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:59:43.880326 6352 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:43.880499 6352 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:43.880944 6352 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:43.882161 6164 ts_manager.cc:194] Registered new tserver with Master: 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:43.883224 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:41139
W20240503 07:59:43.978250 6355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:43.978421 6355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:43.978464 6355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:43.981341 6355 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:43.981392 6355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:43.981472 6355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:43.984848 6355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.6355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:43.985572 6355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:43.986512 6355 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:43.994256 6360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.994724 6361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:43.995265 6363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:43.996310 6355 server_base.cc:1034] running on GCE node
I20240503 07:59:43.996690 6355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:43.997195 6355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:43.998349 6355 hybrid_clock.cc:648] HybridClock initialized: now 1714723183998329 us; error 33 us; skew 500 ppm
I20240503 07:59:44.000041 6355 webserver.cc:460] Webserver started at http://127.0.99.130:34341/ using document root <none> and password file <none>
I20240503 07:59:44.000530 6355 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:44.000583 6355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:44.000736 6355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:44.002259 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-7b05"
I20240503 07:59:44.002686 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-7b05"
I20240503 07:59:44.005620 6355 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:44.007560 6369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.008461 6355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:44.008549 6355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
format_stamp: "Formatted at 2024-05-03 07:59:43 on dist-test-slave-7b05"
I20240503 07:59:44.008630 6355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:44.019721 6355 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:44.020172 6355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:44.020298 6355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:44.020718 6355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:44.021483 6355 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:44.021526 6355 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.021569 6355 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:44.021597 6355 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.028789 6355 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:34351
I20240503 07:59:44.028827 6482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:34351 every 8 connection(s)
I20240503 07:59:44.029582 6355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:44.034301 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 6355
I20240503 07:59:44.034404 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:44.036170 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42987
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:59:44.036995 6483 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:44.037199 6483 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:44.037710 6483 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:44.038578 6164 ts_manager.cc:194] Registered new tserver with Master: 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:44.039086 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:56789
W20240503 07:59:44.134594 6486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:44.134757 6486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:44.134812 6486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:44.137668 6486 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:44.137717 6486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:44.137814 6486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:44.141155 6486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.6486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:44.141851 6486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:44.142781 6486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:44.150331 6492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.150873 6491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.151995 6494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:44.152710 6486 server_base.cc:1034] running on GCE node
I20240503 07:59:44.153093 6486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:44.153575 6486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:44.154707 6486 hybrid_clock.cc:648] HybridClock initialized: now 1714723184154699 us; error 24 us; skew 500 ppm
I20240503 07:59:44.156337 6486 webserver.cc:460] Webserver started at http://127.0.99.131:40873/ using document root <none> and password file <none>
I20240503 07:59:44.156824 6486 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:44.156878 6486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:44.157024 6486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:44.158543 6486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.158963 6486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.161912 6486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:44.163709 6500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.164566 6486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:44.164674 6486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.164743 6486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:44.182272 6486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:44.182741 6486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:44.182871 6486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:44.183303 6486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:44.184042 6486 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:44.184087 6486 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.184132 6486 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:44.184159 6486 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.191429 6486 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34099
I20240503 07:59:44.191474 6613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34099 every 8 connection(s)
I20240503 07:59:44.192206 6486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:44.199823 6614 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:44.200001 6614 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:44.200418 6614 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:44.200999 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 6486
I20240503 07:59:44.201133 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:44.201314 6164 ts_manager.cc:194] Registered new tserver with Master: 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:44.201927 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:58893
I20240503 07:59:44.203209 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42987
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 07:59:44.299626 6617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:44.299799 6617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:44.299857 6617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:44.302742 6617 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:44.302798 6617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:44.302881 6617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:44.306226 6617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.6617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:44.306929 6617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:44.307881 6617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:44.315887 6622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.316368 6623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.316923 6625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:44.317842 6617 server_base.cc:1034] running on GCE node
I20240503 07:59:44.318162 6617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:44.318639 6617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:44.319775 6617 hybrid_clock.cc:648] HybridClock initialized: now 1714723184319755 us; error 39 us; skew 500 ppm
I20240503 07:59:44.321375 6617 webserver.cc:460] Webserver started at http://127.0.99.132:38373/ using document root <none> and password file <none>
I20240503 07:59:44.321851 6617 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:44.321903 6617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:44.322052 6617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:44.323486 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e9976bc168264eaeba1d180fb38b5113"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.323899 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e9976bc168264eaeba1d180fb38b5113"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.326776 6617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:44.328626 6631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.329505 6617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:44.329602 6617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e9976bc168264eaeba1d180fb38b5113"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.329669 6617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:44.338050 6617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:44.338490 6617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:44.338618 6617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:44.339023 6617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:44.339746 6617 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:44.339790 6617 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.339834 6617 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:44.339860 6617 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.347260 6617 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43367
I20240503 07:59:44.347302 6744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43367 every 8 connection(s)
I20240503 07:59:44.348073 6617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:44.355623 6745 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:44.355808 6745 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:44.356254 6745 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:44.357147 6162 ts_manager.cc:194] Registered new tserver with Master: e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:44.357611 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 6617
I20240503 07:59:44.357717 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:59:44.357690 6162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:48909
I20240503 07:59:44.359580 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42987
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 07:59:44.460336 6748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:44.460526 6748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:44.460575 6748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:44.463469 6748 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:44.463522 6748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:44.463593 6748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:44.466977 6748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.6748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:44.467747 6748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:44.468788 6748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:44.477034 6756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.477815 6753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.478533 6754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:44.479219 6748 server_base.cc:1034] running on GCE node
I20240503 07:59:44.479544 6748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:44.480051 6748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:44.481185 6748 hybrid_clock.cc:648] HybridClock initialized: now 1714723184481174 us; error 22 us; skew 500 ppm
I20240503 07:59:44.482864 6748 webserver.cc:460] Webserver started at http://127.0.99.133:36927/ using document root <none> and password file <none>
I20240503 07:59:44.483353 6748 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:44.483402 6748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:44.483592 6748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:44.485144 6748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "61a9a0a13deb4d72b73f8623f6576766"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.485570 6748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "61a9a0a13deb4d72b73f8623f6576766"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.488452 6748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:44.490418 6762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.491295 6748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:44.491389 6748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "61a9a0a13deb4d72b73f8623f6576766"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.491458 6748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:44.511554 6748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:44.512156 6748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:44.512300 6748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:44.512804 6748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:44.513736 6748 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:44.513780 6748 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.513830 6748 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:44.513856 6748 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.521621 6748 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41285
I20240503 07:59:44.521682 6875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41285 every 8 connection(s)
I20240503 07:59:44.524587 6748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:44.534368 6876 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:44.534480 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 6748
I20240503 07:59:44.534600 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:59:44.534618 6876 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:44.535207 6876 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:44.536127 6162 ts_manager.cc:194] Registered new tserver with Master: 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285)
I20240503 07:59:44.536721 6162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:59707
I20240503 07:59:44.537294 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42987
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 07:59:44.636555 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:44.636744 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:44.636804 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:44.639714 6879 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:44.639773 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:44.639858 6879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:44.643929 6879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.6879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:44.644796 6879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:44.646009 6879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:44.654112 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.654959 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.655648 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:44.656245 6879 server_base.cc:1034] running on GCE node
I20240503 07:59:44.656546 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:44.657014 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:44.658180 6879 hybrid_clock.cc:648] HybridClock initialized: now 1714723184658167 us; error 25 us; skew 500 ppm
I20240503 07:59:44.659776 6879 webserver.cc:460] Webserver started at http://127.0.99.134:40691/ using document root <none> and password file <none>
I20240503 07:59:44.660254 6879 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:44.660305 6879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:44.660462 6879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:44.662292 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "66f9a693c3b3409dbcce8f3ee500bf5e"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.662746 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "66f9a693c3b3409dbcce8f3ee500bf5e"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.665874 6879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:44.667892 6893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.668821 6879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:44.668927 6879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "66f9a693c3b3409dbcce8f3ee500bf5e"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.669003 6879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:44.688951 6879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:44.689463 6879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:44.689579 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:44.689992 6879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:44.690717 6879 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:44.690759 6879 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.690794 6879 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:44.690814 6879 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.698604 6879 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42851
I20240503 07:59:44.698657 7006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42851 every 8 connection(s)
I20240503 07:59:44.699460 6879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:44.702040 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 6879
I20240503 07:59:44.702144 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:59:44.704104 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42987
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:59:44.707656 7007 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:44.707832 7007 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:44.708281 7007 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:44.709203 6162 ts_manager.cc:194] Registered new tserver with Master: 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851)
I20240503 07:59:44.709890 6162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:55579
W20240503 07:59:44.802768 7010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:44.802947 7010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:44.802987 7010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:44.805826 7010 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:44.805876 7010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:44.805941 7010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:44.809378 7010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.7010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:44.810138 7010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:44.811089 7010 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:44.818881 7015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.819444 7018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.820331 7016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:44.821015 7010 server_base.cc:1034] running on GCE node
I20240503 07:59:44.821357 7010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:44.821808 7010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:44.822950 7010 hybrid_clock.cc:648] HybridClock initialized: now 1714723184822941 us; error 22 us; skew 500 ppm
I20240503 07:59:44.824532 7010 webserver.cc:460] Webserver started at http://127.0.99.135:36953/ using document root <none> and password file <none>
I20240503 07:59:44.825021 7010 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:44.825101 7010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:44.825268 7010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:44.826732 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "cae9627a7600448faa8fcb7989b307ee"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.827143 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "cae9627a7600448faa8fcb7989b307ee"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.830052 7010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 07:59:44.831915 7024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.832804 7010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:44.832918 7010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "cae9627a7600448faa8fcb7989b307ee"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.832990 7010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:44.852507 7010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:44.852993 7010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:44.853140 7010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:44.853595 7010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:44.854357 7010 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:44.854400 7010 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.854446 7010 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:44.854470 7010 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.861954 7010 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:35381
I20240503 07:59:44.861999 7137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:35381 every 8 connection(s)
I20240503 07:59:44.862716 7010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:44.868820 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 7010
I20240503 07:59:44.868950 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:59:44.870106 7138 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:44.870280 7138 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:44.870800 7138 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:44.870944 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42987
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:59:44.871739 6164 ts_manager.cc:194] Registered new tserver with Master: cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381)
I20240503 07:59:44.872367 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:43183
I20240503 07:59:44.885042 6352 heartbeater.cc:503] Master 127.0.99.190:42987 was elected leader, sending a full tablet report...
W20240503 07:59:44.968955 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:44.969161 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:44.969206 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:44.972051 7141 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:44.972101 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:44.972172 7141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:59:44.975559 7141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.7141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:44.976243 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:44.977221 7141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:44.985029 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.985551 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:44.986518 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:44.987175 7141 server_base.cc:1034] running on GCE node
I20240503 07:59:44.987498 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:44.987967 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:44.989114 7141 hybrid_clock.cc:648] HybridClock initialized: now 1714723184989101 us; error 25 us; skew 500 ppm
I20240503 07:59:44.990754 7141 webserver.cc:460] Webserver started at http://127.0.99.136:45605/ using document root <none> and password file <none>
I20240503 07:59:44.991230 7141 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:44.991281 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:44.991438 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:44.993214 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "713b76caded84f269397165703739870"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.993667 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "713b76caded84f269397165703739870"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.996655 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:44.998514 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:44.999372 7141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:44.999457 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "713b76caded84f269397165703739870"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:44.999531 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:45.011837 7141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:45.012297 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:45.012423 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:45.012853 7141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:45.013664 7141 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:45.013706 7141 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.013744 7141 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:45.013772 7141 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:45.021118 7141 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:38867
I20240503 07:59:45.021163 7268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:38867 every 8 connection(s)
I20240503 07:59:45.021893 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:59:45.025487 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 7141
I20240503 07:59:45.025600 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 07:59:45.029655 7269 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:45.029834 7269 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:45.030257 7269 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:45.031113 6164 ts_manager.cc:194] Registered new tserver with Master: 713b76caded84f269397165703739870 (127.0.99.136:38867)
I20240503 07:59:45.031740 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:53479
I20240503 07:59:45.039336 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:59:45.040620 6483 heartbeater.cc:503] Master 127.0.99.190:42987 was elected leader, sending a full tablet report...
I20240503 07:59:45.047304 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:59:45.047369 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 6748
I20240503 07:59:45.052004 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 6879
I20240503 07:59:45.056494 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 7010
I20240503 07:59:45.061582 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 7141
I20240503 07:59:45.203868 6614 heartbeater.cc:503] Master 127.0.99.190:42987 was elected leader, sending a full tablet report...
I20240503 07:59:45.359521 6745 heartbeater.cc:503] Master 127.0.99.190:42987 was elected leader, sending a full tablet report...
I20240503 07:59:48.817047 6164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:48.835510 6548 tablet_service.cc:1467] Processing CreateTablet for tablet cd0045edfdb441f2ae6920b8a97555bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.835510 6547 tablet_service.cc:1467] Processing CreateTablet for tablet 2f40fae73f8f4c63b2d656ac880799e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.835856 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 2f40fae73f8f4c63b2d656ac880799e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.836448 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f40fae73f8f4c63b2d656ac880799e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.836592 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f40fae73f8f4c63b2d656ac880799e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.837308 6285 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb6518dced248a88edde5cbbb25350d (DEFAULT_TABLE table=rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.837530 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb6518dced248a88edde5cbbb25350d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.837584 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd0045edfdb441f2ae6920b8a97555bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.838563 6416 tablet_service.cc:1467] Processing CreateTablet for tablet 2f40fae73f8f4c63b2d656ac880799e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.838563 6417 tablet_service.cc:1467] Processing CreateTablet for tablet cd0045edfdb441f2ae6920b8a97555bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.838804 6415 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb6518dced248a88edde5cbbb25350d (DEFAULT_TABLE table=rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.839362 6416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f40fae73f8f4c63b2d656ac880799e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.839634 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb6518dced248a88edde5cbbb25350d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.840566 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd0045edfdb441f2ae6920b8a97555bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.843250 7289 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:48.844929 7290 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:48.846067 7289 tablet_bootstrap.cc:654] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.846431 7290 tablet_bootstrap.cc:654] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.846833 7289 log.cc:824] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:48.846994 7290 log.cc:824] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:48.848362 7290 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:48.848362 7289 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:48.848469 7291 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:48.848520 7289 ts_tablet_manager.cc:1367] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:48.848559 7290 ts_tablet_manager.cc:1367] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:48.850483 6677 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb6518dced248a88edde5cbbb25350d (DEFAULT_TABLE table=rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.851279 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb6518dced248a88edde5cbbb25350d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.852144 7291 tablet_bootstrap.cc:654] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.852032 7290 raft_consensus.cc:348] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.852221 7290 raft_consensus.cc:374] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.852164 7289 raft_consensus.cc:348] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:48.852296 7290 raft_consensus.cc:729] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.852422 7289 raft_consensus.cc:374] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.852555 7289 raft_consensus.cc:729] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.852711 7290 consensus_queue.cc:260] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.853049 7289 consensus_queue.cc:260] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:48.853447 7291 log.cc:824] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:48.853551 7290 ts_tablet_manager.cc:1398] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:48.853895 7290 tablet_bootstrap.cc:492] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:48.853940 7289 ts_tablet_manager.cc:1398] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:48.854291 7289 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:48.854766 7290 tablet_bootstrap.cc:654] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.855213 7289 tablet_bootstrap.cc:654] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.850483 6679 tablet_service.cc:1467] Processing CreateTablet for tablet cd0045edfdb441f2ae6920b8a97555bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.856236 7289 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:48.856298 7289 ts_tablet_manager.cc:1367] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.856308 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd0045edfdb441f2ae6920b8a97555bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.856784 7289 raft_consensus.cc:348] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.856865 7289 raft_consensus.cc:374] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.856894 7289 raft_consensus.cc:729] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.856997 7289 consensus_queue.cc:260] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.857257 7289 ts_tablet_manager.cc:1398] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.857440 7290 tablet_bootstrap.cc:492] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:48.857494 7290 ts_tablet_manager.cc:1367] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:48.858021 7290 raft_consensus.cc:348] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.858188 7290 raft_consensus.cc:374] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.858259 7290 raft_consensus.cc:729] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.858443 7290 consensus_queue.cc:260] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.858829 7290 ts_tablet_manager.cc:1398] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.859948 7291 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:48.860131 7291 ts_tablet_manager.cc:1367] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240503 07:59:48.860332 7300 tablet_bootstrap.cc:492] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:48.861853 7300 tablet_bootstrap.cc:654] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.862396 7300 log.cc:824] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:48.862615 7291 raft_consensus.cc:348] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:48.862820 7291 raft_consensus.cc:374] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.862910 7291 raft_consensus.cc:729] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.863317 7291 consensus_queue.cc:260] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:48.863960 7300 tablet_bootstrap.cc:492] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:48.863996 7291 ts_tablet_manager.cc:1398] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:48.864112 7300 ts_tablet_manager.cc:1367] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:48.864217 7291 tablet_bootstrap.cc:492] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:48.865093 7291 tablet_bootstrap.cc:654] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.866079 7291 tablet_bootstrap.cc:492] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:48.866164 7291 ts_tablet_manager.cc:1367] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.866631 7291 raft_consensus.cc:348] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.866715 7291 raft_consensus.cc:374] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.866767 7291 raft_consensus.cc:729] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.866906 7291 consensus_queue.cc:260] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.867147 7291 ts_tablet_manager.cc:1398] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.867091 7300 raft_consensus.cc:348] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.867303 7300 raft_consensus.cc:374] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.867344 7291 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:48.867372 7300 raft_consensus.cc:729] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.867722 7300 consensus_queue.cc:260] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.868139 7291 tablet_bootstrap.cc:654] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.869210 7300 ts_tablet_manager.cc:1398] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:48.869441 7300 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:48.869679 7291 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:48.869746 7291 ts_tablet_manager.cc:1367] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.870091 7291 raft_consensus.cc:348] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.870173 7291 raft_consensus.cc:374] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.870203 7291 raft_consensus.cc:729] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.870273 7300 tablet_bootstrap.cc:654] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.870357 7291 consensus_queue.cc:260] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.870661 7291 ts_tablet_manager.cc:1398] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:48.871941 7300 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:48.872058 7300 ts_tablet_manager.cc:1367] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:48.872538 7300 raft_consensus.cc:348] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:48.872685 7300 raft_consensus.cc:374] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.872750 7300 raft_consensus.cc:729] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.872885 7300 consensus_queue.cc:260] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:48.873155 7300 ts_tablet_manager.cc:1398] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:48.876093 6353 tablet.cc:2369] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:48.946094 6615 tablet.cc:2369] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:48.972453 7295 raft_consensus.cc:482] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:48.972615 7295 raft_consensus.cc:504] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.973629 7295 leader_election.cc:288] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:48.977079 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:48.977311 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:48.977350 6306 raft_consensus.cc:2454] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:48.977546 6437 raft_consensus.cc:2454] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:48.977715 6502 leader_election.cc:302] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:48.977931 7295 raft_consensus.cc:2781] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:48.977998 7295 raft_consensus.cc:482] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:48.978044 7295 raft_consensus.cc:3037] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.978821 7295 raft_consensus.cc:504] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.979084 7295 leader_election.cc:288] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:48.979369 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:48.979393 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:48.979470 6437 raft_consensus.cc:3037] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.979475 6306 raft_consensus.cc:3037] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.980193 6306 raft_consensus.cc:2454] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:48.980284 6437 raft_consensus.cc:2454] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:48.980473 6502 leader_election.cc:302] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:48.980615 7295 raft_consensus.cc:2781] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:48.980799 7295 raft_consensus.cc:686] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:48.981108 7295 consensus_queue.cc:237] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:48.983418 6162 catalog_manager.cc:5484] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:49.033470 6484 tablet.cc:2369] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:49.034876 7295 raft_consensus.cc:482] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.034982 7295 raft_consensus.cc:504] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.035459 7295 leader_election.cc:288] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:49.035770 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0045edfdb441f2ae6920b8a97555bb" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:49.035897 6437 raft_consensus.cc:2454] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:49.036167 6501 leader_election.cc:302] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:49.036366 7295 raft_consensus.cc:2781] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.036428 7295 raft_consensus.cc:482] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.036502 7295 raft_consensus.cc:3037] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.037393 7295 raft_consensus.cc:504] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.037683 7295 leader_election.cc:288] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:49.037964 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0045edfdb441f2ae6920b8a97555bb" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:49.038057 6437 raft_consensus.cc:3037] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.038749 6437 raft_consensus.cc:2454] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:49.038996 6501 leader_election.cc:302] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:49.039040 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0045edfdb441f2ae6920b8a97555bb" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:49.039043 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0045edfdb441f2ae6920b8a97555bb" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:49.039148 7295 raft_consensus.cc:2781] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.039253 7295 raft_consensus.cc:686] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:49.039275 6698 raft_consensus.cc:3037] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.039448 7295 consensus_queue.cc:237] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.040269 6698 raft_consensus.cc:2454] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:49.041383 6162 catalog_manager.cc:5484] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:49.101639 6746 tablet.cc:2369] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:49.119321 7304 raft_consensus.cc:482] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.119455 7304 raft_consensus.cc:504] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:49.120432 7304 leader_election.cc:288] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:49.123701 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb6518dced248a88edde5cbbb25350d" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:49.123725 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb6518dced248a88edde5cbbb25350d" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:49.123836 6306 raft_consensus.cc:2454] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:49.123836 6437 raft_consensus.cc:2454] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:49.124486 6633 leader_election.cc:302] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:49.124712 7304 raft_consensus.cc:2781] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.124776 7304 raft_consensus.cc:482] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.124819 7304 raft_consensus.cc:3037] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.125612 7304 raft_consensus.cc:504] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:49.125891 7304 leader_election.cc:288] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:49.126169 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb6518dced248a88edde5cbbb25350d" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:49.126179 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb6518dced248a88edde5cbbb25350d" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:49.126271 6306 raft_consensus.cc:3037] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.126271 6437 raft_consensus.cc:3037] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.126950 6306 raft_consensus.cc:2454] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:49.127024 6437 raft_consensus.cc:2454] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:49.127203 6633 leader_election.cc:302] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:49.127362 7304 raft_consensus.cc:2781] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.127543 7304 raft_consensus.cc:686] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Becoming Leader. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Running, Role: LEADER
I20240503 07:59:49.127844 7304 consensus_queue.cc:237] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:49.130429 6162 catalog_manager.cc:5484] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9976bc168264eaeba1d180fb38b5113 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:49.289610 7316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:49.289801 7316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:49.292650 7316 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 07:59:49.368844 6306 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.369833 6437 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.370429 6698 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.371018 7309 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.371171 6436 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.371985 7295 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.372256 7295 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.372532 7295 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.373674 6437 raft_consensus.cc:1261] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.373988 6306 raft_consensus.cc:1261] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.376013 7314 consensus_queue.cc:1035] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.378835 7314 consensus_queue.cc:1035] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.381280 7345 mvcc.cc:205] Tried to move back new op lower bound from 7023506183648071680 to 7023506182306856960. Current Snapshot: MvccSnapshot[applied={T|T < 7023506183648071680}]
I20240503 07:59:49.392300 7356 mvcc.cc:205] Tried to move back new op lower bound from 7023506183665123328 to 7023506182669803520. Current Snapshot: MvccSnapshot[applied={T|T < 7023506183650746368}]
I20240503 07:59:49.392542 7351 mvcc.cc:205] Tried to move back new op lower bound from 7023506183648071680 to 7023506182306856960. Current Snapshot: MvccSnapshot[applied={T|T < 7023506183648071680}]
I20240503 07:59:49.395701 7346 mvcc.cc:205] Tried to move back new op lower bound from 7023506183648858112 to 7023506182068813824. Current Snapshot: MvccSnapshot[applied={T|T < 7023506183648858112}]
INSERT report
rows total: 2000
time total: 103.534 ms
time per row: 0.0517672 ms
I20240503 07:59:49.435976 6162 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:49.440759 6679 tablet_service.cc:1467] Processing CreateTablet for tablet 08511a1b1eab4da9b810283d12cccee3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.440831 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 08511a1b1eab4da9b810283d12cccee3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.441049 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08511a1b1eab4da9b810283d12cccee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.441140 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08511a1b1eab4da9b810283d12cccee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.441155 6285 tablet_service.cc:1467] Processing CreateTablet for tablet 4418f348094548daa128685753415583 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.441358 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4418f348094548daa128685753415583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.441704 6417 tablet_service.cc:1467] Processing CreateTablet for tablet ffd6a5145bf64148b1c9db148905f92b (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.441949 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd6a5145bf64148b1c9db148905f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.443500 7381 tablet_bootstrap.cc:492] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:49.443792 6285 tablet_service.cc:1467] Processing CreateTablet for tablet ffd6a5145bf64148b1c9db148905f92b (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.443995 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd6a5145bf64148b1c9db148905f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.444375 7381 tablet_bootstrap.cc:654] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.444317 6547 tablet_service.cc:1467] Processing CreateTablet for tablet 08511a1b1eab4da9b810283d12cccee3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.444567 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08511a1b1eab4da9b810283d12cccee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.445303 6546 tablet_service.cc:1467] Processing CreateTablet for tablet ffd6a5145bf64148b1c9db148905f92b (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.445403 7381 tablet_bootstrap.cc:492] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:49.445472 7381 ts_tablet_manager.cc:1367] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.445525 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd6a5145bf64148b1c9db148905f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.445863 7381 raft_consensus.cc:348] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.445952 7381 raft_consensus.cc:374] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.445979 7381 raft_consensus.cc:729] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.446084 7381 consensus_queue.cc:260] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.446115 7380 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:49.446336 7381 ts_tablet_manager.cc:1398] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.446744 6548 tablet_service.cc:1467] Processing CreateTablet for tablet 4418f348094548daa128685753415583 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.446859 7380 tablet_bootstrap.cc:654] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.446935 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4418f348094548daa128685753415583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.447837 7379 tablet_bootstrap.cc:492] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:49.448628 7379 tablet_bootstrap.cc:654] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.449327 6679 tablet_service.cc:1467] Processing CreateTablet for tablet 4418f348094548daa128685753415583 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.449537 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4418f348094548daa128685753415583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.450930 7383 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:49.451537 7383 tablet_bootstrap.cc:654] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.452571 7380 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:49.452572 7383 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:49.452651 7383 ts_tablet_manager.cc:1367] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.452688 7380 ts_tablet_manager.cc:1367] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:49.452752 7379 tablet_bootstrap.cc:492] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:49.452831 7379 ts_tablet_manager.cc:1367] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:49.453091 7383 raft_consensus.cc:348] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.453115 7380 raft_consensus.cc:348] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.453209 7380 raft_consensus.cc:374] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.453258 7380 raft_consensus.cc:729] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.453267 7379 raft_consensus.cc:348] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.453325 7383 raft_consensus.cc:374] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.453358 7379 raft_consensus.cc:374] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.453363 7383 raft_consensus.cc:729] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.453387 7379 raft_consensus.cc:729] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.453400 7380 consensus_queue.cc:260] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.453488 7383 consensus_queue.cc:260] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.453508 7379 consensus_queue.cc:260] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.453768 7380 ts_tablet_manager.cc:1398] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.453799 7383 ts_tablet_manager.cc:1398] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.453895 7380 tablet_bootstrap.cc:492] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:49.453938 7383 tablet_bootstrap.cc:492] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:49.454561 7379 ts_tablet_manager.cc:1398] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.454713 7379 tablet_bootstrap.cc:492] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:49.454721 7383 tablet_bootstrap.cc:654] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.454674 7380 tablet_bootstrap.cc:654] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.455471 7379 tablet_bootstrap.cc:654] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.455716 7383 tablet_bootstrap.cc:492] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:49.455781 7383 ts_tablet_manager.cc:1367] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.455893 7380 tablet_bootstrap.cc:492] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:49.455960 7380 ts_tablet_manager.cc:1367] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:49.456127 7383 raft_consensus.cc:348] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.456193 7383 raft_consensus.cc:374] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.456212 7383 raft_consensus.cc:729] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.456303 7383 consensus_queue.cc:260] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.456351 7380 raft_consensus.cc:348] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.456446 7380 raft_consensus.cc:374] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.456482 7380 raft_consensus.cc:729] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.456589 7383 ts_tablet_manager.cc:1398] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.456585 7380 consensus_queue.cc:260] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.456734 7383 tablet_bootstrap.cc:492] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:49.456799 7380 ts_tablet_manager.cc:1398] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.457504 7383 tablet_bootstrap.cc:654] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.457500 7379 tablet_bootstrap.cc:492] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:49.457576 7379 ts_tablet_manager.cc:1367] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:49.458000 7379 raft_consensus.cc:348] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.458107 7379 raft_consensus.cc:374] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.458168 7379 raft_consensus.cc:729] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.458309 7379 consensus_queue.cc:260] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.458563 7383 tablet_bootstrap.cc:492] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:49.458559 7379 ts_tablet_manager.cc:1398] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:49.458631 7383 ts_tablet_manager.cc:1367] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:49.458701 7379 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:49.458962 7383 raft_consensus.cc:348] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.459033 7383 raft_consensus.cc:374] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.459060 7383 raft_consensus.cc:729] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.459205 7383 consensus_queue.cc:260] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.459419 7383 ts_tablet_manager.cc:1398] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:49.459826 7379 tablet_bootstrap.cc:654] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.460836 7379 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:49.460964 7379 ts_tablet_manager.cc:1367] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:49.461489 7379 raft_consensus.cc:348] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.461585 7379 raft_consensus.cc:374] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.461611 7379 raft_consensus.cc:729] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.461711 7379 consensus_queue.cc:260] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.461943 7379 ts_tablet_manager.cc:1398] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:49.476857 7314 raft_consensus.cc:482] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.476960 7314 raft_consensus.cc:504] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.477442 7314 leader_election.cc:288] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:49.477814 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4418f348094548daa128685753415583" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:49.477945 6305 raft_consensus.cc:2454] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:49.478201 6633 leader_election.cc:302] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:49.478405 7314 raft_consensus.cc:2781] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.478459 7314 raft_consensus.cc:482] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.478494 7314 raft_consensus.cc:3037] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.479442 7314 raft_consensus.cc:504] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.479727 7314 leader_election.cc:288] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Requested vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:49.479996 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4418f348094548daa128685753415583" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:49.480095 6305 raft_consensus.cc:3037] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.480959 6305 raft_consensus.cc:2454] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:49.481112 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4418f348094548daa128685753415583" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:49.481124 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4418f348094548daa128685753415583" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:49.481251 6633 leader_election.cc:302] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:49.481312 6567 raft_consensus.cc:3037] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.481395 7314 raft_consensus.cc:2781] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.481472 7314 raft_consensus.cc:686] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Becoming Leader. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Running, Role: LEADER
I20240503 07:59:49.481592 7314 consensus_queue.cc:237] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.482249 6567 raft_consensus.cc:2454] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:49.483057 6159 catalog_manager.cc:5484] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9976bc168264eaeba1d180fb38b5113 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:49.505084 7355 raft_consensus.cc:482] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.505177 7355 raft_consensus.cc:504] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.506112 7355 leader_election.cc:288] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:49.509717 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd6a5145bf64148b1c9db148905f92b" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:49.509847 6437 raft_consensus.cc:2454] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:49.509841 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd6a5145bf64148b1c9db148905f92b" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:49.509965 6567 raft_consensus.cc:2454] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:49.510138 6239 leader_election.cc:302] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:49.510330 7355 raft_consensus.cc:2781] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.510385 7355 raft_consensus.cc:482] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.510424 7355 raft_consensus.cc:3037] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.511142 7355 raft_consensus.cc:504] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.511443 7355 leader_election.cc:288] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:49.511723 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd6a5145bf64148b1c9db148905f92b" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:49.511724 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd6a5145bf64148b1c9db148905f92b" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:49.511816 6567 raft_consensus.cc:3037] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.511839 6437 raft_consensus.cc:3037] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.512483 6567 raft_consensus.cc:2454] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:49.512559 6437 raft_consensus.cc:2454] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:49.512735 6241 leader_election.cc:302] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:49.512883 7355 raft_consensus.cc:2781] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.512962 7355 raft_consensus.cc:686] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Becoming Leader. State: Replica: 253dd189c3c542248402cba222394f43, State: Running, Role: LEADER
I20240503 07:59:49.513235 7355 consensus_queue.cc:237] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.515077 6159 catalog_manager.cc:5484] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 253dd189c3c542248402cba222394f43 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:49.660876 7355 raft_consensus.cc:482] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.660996 7355 raft_consensus.cc:504] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.661468 7355 leader_election.cc:288] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:49.662128 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:49.662276 6567 raft_consensus.cc:2454] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:49.662544 6241 leader_election.cc:302] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:49.663131 7355 raft_consensus.cc:2781] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.663200 7355 raft_consensus.cc:482] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.663241 7355 raft_consensus.cc:3037] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.664156 7355 raft_consensus.cc:504] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.664445 7355 leader_election.cc:288] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:49.664804 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:49.664870 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:49.664920 6698 raft_consensus.cc:2454] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:49.665163 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:49.665278 6567 raft_consensus.cc:3037] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.665925 6567 raft_consensus.cc:2454] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:49.666183 6241 leader_election.cc:302] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters: e9976bc168264eaeba1d180fb38b5113
I20240503 07:59:49.666335 7355 raft_consensus.cc:2781] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.666410 7355 raft_consensus.cc:686] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Becoming Leader. State: Replica: 253dd189c3c542248402cba222394f43, State: Running, Role: LEADER
I20240503 07:59:49.666540 7355 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.667989 6159 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 253dd189c3c542248402cba222394f43 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:49.778075 7397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:49.778277 7397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:49.781168 7397 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 07:59:49.851550 6305 raft_consensus.cc:1261] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.851555 6699 raft_consensus.cc:3037] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.851889 6567 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.852285 6568 raft_consensus.cc:1261] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.852659 6699 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.852830 6437 raft_consensus.cc:1261] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.853312 7314 consensus_queue.cc:1035] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.853554 7350 consensus_queue.cc:1035] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.853544 7355 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.855736 7314 consensus_queue.cc:1035] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.856514 6568 raft_consensus.cc:1261] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.857124 7350 consensus_queue.cc:1035] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.857023 7355 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 98.9531 ms
time per row: 0.0494765 ms
I20240503 07:59:49.920532 6162 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:49.925300 6286 tablet_service.cc:1467] Processing CreateTablet for tablet d9e91e1b70504ac695eae61f72d3c149 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.925563 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e91e1b70504ac695eae61f72d3c149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.925949 6548 tablet_service.cc:1467] Processing CreateTablet for tablet 37c8748a1c9d42b593b2381a1a870d2e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.926175 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c8748a1c9d42b593b2381a1a870d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.926097 6677 tablet_service.cc:1467] Processing CreateTablet for tablet 37c8748a1c9d42b593b2381a1a870d2e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.926385 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c8748a1c9d42b593b2381a1a870d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.927021 6678 tablet_service.cc:1467] Processing CreateTablet for tablet 4925e30c2fb74934bbd5eb7415193c96 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.927244 6678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4925e30c2fb74934bbd5eb7415193c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.927935 6679 tablet_service.cc:1467] Processing CreateTablet for tablet d9e91e1b70504ac695eae61f72d3c149 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.928125 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e91e1b70504ac695eae61f72d3c149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.928282 7380 tablet_bootstrap.cc:492] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:49.928520 7383 tablet_bootstrap.cc:492] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:49.929113 7380 tablet_bootstrap.cc:654] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.929350 7383 tablet_bootstrap.cc:654] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.929842 7379 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:49.930334 7383 tablet_bootstrap.cc:492] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:49.930416 7383 ts_tablet_manager.cc:1367] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:49.930449 6415 tablet_service.cc:1467] Processing CreateTablet for tablet 4925e30c2fb74934bbd5eb7415193c96 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.930579 7379 tablet_bootstrap.cc:654] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.930691 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4925e30c2fb74934bbd5eb7415193c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.930830 7383 raft_consensus.cc:348] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.930933 7383 raft_consensus.cc:374] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.930972 7383 raft_consensus.cc:729] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.931088 7383 consensus_queue.cc:260] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.931331 7383 ts_tablet_manager.cc:1398] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.931406 7379 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:49.931463 7379 ts_tablet_manager.cc:1367] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.931805 7379 raft_consensus.cc:348] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.931883 7379 raft_consensus.cc:374] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.931905 7379 raft_consensus.cc:729] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.931998 7379 consensus_queue.cc:260] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.932222 7379 ts_tablet_manager.cc:1398] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.932667 7381 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:49.932715 6416 tablet_service.cc:1467] Processing CreateTablet for tablet 37c8748a1c9d42b593b2381a1a870d2e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.932946 6416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c8748a1c9d42b593b2381a1a870d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.933442 7381 tablet_bootstrap.cc:654] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.934286 7381 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:49.934371 7381 ts_tablet_manager.cc:1367] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.934720 7381 raft_consensus.cc:348] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.934722 6417 tablet_service.cc:1467] Processing CreateTablet for tablet d9e91e1b70504ac695eae61f72d3c149 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.934860 7381 raft_consensus.cc:374] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.934901 7381 raft_consensus.cc:729] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.934986 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e91e1b70504ac695eae61f72d3c149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.935009 7381 consensus_queue.cc:260] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.935264 7381 ts_tablet_manager.cc:1398] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.935420 7381 tablet_bootstrap.cc:492] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:49.935719 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 4925e30c2fb74934bbd5eb7415193c96 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.935926 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4925e30c2fb74934bbd5eb7415193c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.936156 7381 tablet_bootstrap.cc:654] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.937230 7381 tablet_bootstrap.cc:492] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:49.937321 7381 ts_tablet_manager.cc:1367] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.937696 7379 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:49.937709 7381 raft_consensus.cc:348] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.937791 7381 raft_consensus.cc:374] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.937822 7381 raft_consensus.cc:729] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.937930 7381 consensus_queue.cc:260] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.938172 7381 ts_tablet_manager.cc:1398] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:49.938318 7381 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:49.938489 7379 tablet_bootstrap.cc:654] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.939072 7381 tablet_bootstrap.cc:654] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.939375 7380 tablet_bootstrap.cc:492] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:49.939432 7380 ts_tablet_manager.cc:1367] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20240503 07:59:49.939538 7379 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:49.939599 7379 ts_tablet_manager.cc:1367] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.939800 7380 raft_consensus.cc:348] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.939882 7380 raft_consensus.cc:374] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.939901 7380 raft_consensus.cc:729] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.939939 7379 raft_consensus.cc:348] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.939990 7380 consensus_queue.cc:260] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:49.940019 7381 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:49.940023 7379 raft_consensus.cc:374] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.940052 7379 raft_consensus.cc:729] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.940079 7381 ts_tablet_manager.cc:1367] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.940145 7379 consensus_queue.cc:260] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.940202 7380 ts_tablet_manager.cc:1398] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:49.940335 7380 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:49.940394 7379 ts_tablet_manager.cc:1398] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.940512 7381 raft_consensus.cc:348] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.940590 7381 raft_consensus.cc:374] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.940618 7381 raft_consensus.cc:729] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.940699 7381 consensus_queue.cc:260] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.940923 7381 ts_tablet_manager.cc:1398] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.941123 7380 tablet_bootstrap.cc:654] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.942134 7380 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:49.942219 7380 ts_tablet_manager.cc:1367] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:49.942582 7380 raft_consensus.cc:348] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.942672 7380 raft_consensus.cc:374] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.942700 7380 raft_consensus.cc:729] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.942819 7380 consensus_queue.cc:260] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.943150 7380 ts_tablet_manager.cc:1398] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:49.943308 7380 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:49.944093 7380 tablet_bootstrap.cc:654] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.945122 7380 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:49.945204 7380 ts_tablet_manager.cc:1367] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:49.945783 7380 raft_consensus.cc:348] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.945883 7380 raft_consensus.cc:374] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.945923 7380 raft_consensus.cc:729] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.946053 7380 consensus_queue.cc:260] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.946302 7380 ts_tablet_manager.cc:1398] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.946681 7359 raft_consensus.cc:482] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.946765 7359 raft_consensus.cc:504] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.947818 7359 leader_election.cc:288] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:49.951468 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:49.951586 6305 raft_consensus.cc:2454] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:49.951745 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:49.951851 6699 raft_consensus.cc:2454] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:49.951920 6371 leader_election.cc:302] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:49.952111 7359 raft_consensus.cc:2781] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.952162 7359 raft_consensus.cc:482] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.952188 7359 raft_consensus.cc:3037] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.952926 7359 raft_consensus.cc:504] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.953226 7359 leader_election.cc:288] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Requested vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:49.953497 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:49.953586 6305 raft_consensus.cc:3037] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.953590 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:49.953711 6699 raft_consensus.cc:3037] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.954296 6305 raft_consensus.cc:2454] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:49.954355 6699 raft_consensus.cc:2454] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:49.954564 6371 leader_election.cc:302] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:49.954795 7359 raft_consensus.cc:2781] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.954959 7359 raft_consensus.cc:686] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Leader. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 07:59:49.955194 7359 consensus_queue.cc:237] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.957074 6157 catalog_manager.cc:5484] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:49.971017 7359 raft_consensus.cc:482] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.971100 7359 raft_consensus.cc:504] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.971377 7359 leader_election.cc:288] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:49.971675 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4925e30c2fb74934bbd5eb7415193c96" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:49.971720 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4925e30c2fb74934bbd5eb7415193c96" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:49.971779 6305 raft_consensus.cc:2454] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:49.971855 6699 raft_consensus.cc:2454] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:49.972020 6371 leader_election.cc:302] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:49.972174 7359 raft_consensus.cc:2781] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.972229 7359 raft_consensus.cc:482] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.972271 7359 raft_consensus.cc:3037] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.972947 7359 raft_consensus.cc:504] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.973248 7359 leader_election.cc:288] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Requested vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:49.973522 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4925e30c2fb74934bbd5eb7415193c96" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:49.973567 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4925e30c2fb74934bbd5eb7415193c96" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:49.973611 6305 raft_consensus.cc:3037] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.973806 6699 raft_consensus.cc:3037] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.974244 6305 raft_consensus.cc:2454] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:49.974463 6371 leader_election.cc:302] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:49.974532 6699 raft_consensus.cc:2454] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:49.974603 7359 raft_consensus.cc:2781] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.974682 7359 raft_consensus.cc:686] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Leader. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 07:59:49.974800 7359 consensus_queue.cc:237] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:49.976150 6157 catalog_manager.cc:5484] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:50.013263 7359 raft_consensus.cc:482] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.013365 7359 raft_consensus.cc:504] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:50.013729 7359 leader_election.cc:288] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:50.014034 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:50.014173 6699 raft_consensus.cc:2454] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:50.014429 6372 leader_election.cc:302] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:50.014578 7359 raft_consensus.cc:2781] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.014629 7359 raft_consensus.cc:482] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.014670 7359 raft_consensus.cc:3037] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.015408 7359 raft_consensus.cc:504] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:50.015729 7359 leader_election.cc:288] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:50.016009 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:50.016121 6699 raft_consensus.cc:3037] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.016981 6699 raft_consensus.cc:2454] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:50.017235 6372 leader_election.cc:302] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:50.017382 7359 raft_consensus.cc:2781] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.017463 7359 raft_consensus.cc:686] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Leader. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 07:59:50.017414 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:50.017495 6566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:50.017561 6567 raft_consensus.cc:2454] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:50.017589 7359 consensus_queue.cc:237] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:50.019078 6157 catalog_manager.cc:5484] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:50.124677 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:50.124871 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:50.127744 7450 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 07:59:50.193387 6305 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.193670 6306 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.193660 6699 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.193946 6698 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.194047 7359 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.194388 7359 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.194586 7359 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.194795 7359 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.196962 6567 raft_consensus.cc:3037] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.198033 6567 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.198767 7359 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.204603 6697 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.204952 7359 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 96.5416 ms
time per row: 0.0482708 ms
I20240503 07:59:50.259582 6162 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:50.264312 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 9e47575b08ad4f54b1f8d35db27d9e58 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.264488 6417 tablet_service.cc:1467] Processing CreateTablet for tablet 9e47575b08ad4f54b1f8d35db27d9e58 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.264611 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e47575b08ad4f54b1f8d35db27d9e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.264755 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e47575b08ad4f54b1f8d35db27d9e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.264983 6548 tablet_service.cc:1467] Processing CreateTablet for tablet 9e47575b08ad4f54b1f8d35db27d9e58 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.265020 6416 tablet_service.cc:1467] Processing CreateTablet for tablet 8a934439594245d79ed5980ff9c03277 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.265244 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e47575b08ad4f54b1f8d35db27d9e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.265256 6416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a934439594245d79ed5980ff9c03277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.266727 7381 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:50.267416 6546 tablet_service.cc:1467] Processing CreateTablet for tablet 885b10f956b54ebaa82614f8a1f979d1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.267478 7381 tablet_bootstrap.cc:654] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.267628 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b10f956b54ebaa82614f8a1f979d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.267732 6679 tablet_service.cc:1467] Processing CreateTablet for tablet 885b10f956b54ebaa82614f8a1f979d1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.267969 7379 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:50.267984 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b10f956b54ebaa82614f8a1f979d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.267954 6548 tablet_service.cc:1467] Processing CreateTablet for tablet 8a934439594245d79ed5980ff9c03277 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.268131 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a934439594245d79ed5980ff9c03277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.268795 7379 tablet_bootstrap.cc:654] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.269846 7380 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:50.267458 7383 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:50.270030 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 885b10f956b54ebaa82614f8a1f979d1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.270226 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b10f956b54ebaa82614f8a1f979d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.270570 7380 tablet_bootstrap.cc:654] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.270572 7383 tablet_bootstrap.cc:654] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.267732 6678 tablet_service.cc:1467] Processing CreateTablet for tablet 8a934439594245d79ed5980ff9c03277 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.271215 6678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a934439594245d79ed5980ff9c03277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.271521 7380 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:50.271536 7383 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:50.271584 7380 ts_tablet_manager.cc:1367] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:50.271605 7383 ts_tablet_manager.cc:1367] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:50.271996 7380 raft_consensus.cc:348] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.272006 7383 raft_consensus.cc:348] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.272106 7383 raft_consensus.cc:374] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.272106 7380 raft_consensus.cc:374] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.272143 7383 raft_consensus.cc:729] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.272143 7380 raft_consensus.cc:729] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.272248 7380 consensus_queue.cc:260] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.272260 7383 consensus_queue.cc:260] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.272522 7380 ts_tablet_manager.cc:1398] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:50.272547 7383 ts_tablet_manager.cc:1398] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:50.272691 7383 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:50.272753 7379 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:50.272815 7379 ts_tablet_manager.cc:1367] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:50.273218 7379 raft_consensus.cc:348] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.273306 7379 raft_consensus.cc:374] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.273335 7379 raft_consensus.cc:729] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.273432 7379 consensus_queue.cc:260] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.273597 7380 tablet_bootstrap.cc:492] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:50.273680 7379 ts_tablet_manager.cc:1398] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:50.273824 7379 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:50.274367 7380 tablet_bootstrap.cc:654] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.274621 7379 tablet_bootstrap.cc:654] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.275202 7380 tablet_bootstrap.cc:492] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:50.275270 7380 ts_tablet_manager.cc:1367] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:50.275584 7381 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:50.275663 7381 ts_tablet_manager.cc:1367] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:50.275653 7380 raft_consensus.cc:348] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:50.275728 7380 raft_consensus.cc:374] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.275756 7380 raft_consensus.cc:729] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.275784 7379 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:50.275848 7379 ts_tablet_manager.cc:1367] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:50.275877 7380 consensus_queue.cc:260] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:50.276036 7381 raft_consensus.cc:348] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.276124 7381 raft_consensus.cc:374] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.276154 7381 raft_consensus.cc:729] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.276166 7380 ts_tablet_manager.cc:1398] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:50.276224 7379 raft_consensus.cc:348] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.276269 7381 consensus_queue.cc:260] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.276310 7379 raft_consensus.cc:374] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.276340 7379 raft_consensus.cc:729] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.276437 7379 consensus_queue.cc:260] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.276504 7381 ts_tablet_manager.cc:1398] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:50.276654 7381 tablet_bootstrap.cc:492] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:50.276674 7379 ts_tablet_manager.cc:1398] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:50.277514 7381 tablet_bootstrap.cc:654] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.278501 7381 tablet_bootstrap.cc:492] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:50.278610 7383 tablet_bootstrap.cc:654] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.278579 7381 ts_tablet_manager.cc:1367] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:50.279247 7381 raft_consensus.cc:348] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:50.279333 7381 raft_consensus.cc:374] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.279368 7381 raft_consensus.cc:729] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.279500 7381 consensus_queue.cc:260] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:50.279682 7383 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:50.279753 7383 ts_tablet_manager.cc:1367] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 07:59:50.279814 7381 ts_tablet_manager.cc:1398] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:50.280143 7383 raft_consensus.cc:348] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.280251 7383 raft_consensus.cc:374] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.280282 7383 raft_consensus.cc:729] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.280411 7383 consensus_queue.cc:260] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.280666 7383 ts_tablet_manager.cc:1398] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:50.280875 7383 tablet_bootstrap.cc:492] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:50.281692 7383 tablet_bootstrap.cc:654] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.282524 7383 tablet_bootstrap.cc:492] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:50.282603 7383 ts_tablet_manager.cc:1367] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:50.282934 7383 raft_consensus.cc:348] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:50.283008 7383 raft_consensus.cc:374] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.283027 7383 raft_consensus.cc:729] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.283113 7383 consensus_queue.cc:260] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:50.283344 7383 ts_tablet_manager.cc:1398] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:50.287437 7355 raft_consensus.cc:482] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.287505 7355 raft_consensus.cc:504] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.287791 7355 leader_election.cc:288] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:50.288041 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e47575b08ad4f54b1f8d35db27d9e58" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:50.288062 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e47575b08ad4f54b1f8d35db27d9e58" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:50.288130 6567 raft_consensus.cc:2454] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:50.288156 6437 raft_consensus.cc:2454] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:50.288372 6241 leader_election.cc:302] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:50.288513 7355 raft_consensus.cc:2781] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.288573 7355 raft_consensus.cc:482] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.288614 7355 raft_consensus.cc:3037] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.289335 7355 raft_consensus.cc:504] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.289618 7355 leader_election.cc:288] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Requested vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:50.289855 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e47575b08ad4f54b1f8d35db27d9e58" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:50.289877 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e47575b08ad4f54b1f8d35db27d9e58" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:50.289958 6567 raft_consensus.cc:3037] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.289958 6437 raft_consensus.cc:3037] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.290752 6567 raft_consensus.cc:2454] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:50.290752 6437 raft_consensus.cc:2454] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:50.290992 6241 leader_election.cc:302] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:50.291183 7355 raft_consensus.cc:2781] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.291252 7355 raft_consensus.cc:686] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Becoming Leader. State: Replica: 253dd189c3c542248402cba222394f43, State: Running, Role: LEADER
I20240503 07:59:50.291373 7355 consensus_queue.cc:237] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.292596 6162 catalog_manager.cc:5484] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 253dd189c3c542248402cba222394f43 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:50.382253 7347 raft_consensus.cc:482] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.382370 7347 raft_consensus.cc:504] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.382758 7347 leader_election.cc:288] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:50.383086 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b10f956b54ebaa82614f8a1f979d1" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:50.383170 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b10f956b54ebaa82614f8a1f979d1" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:50.383213 6699 raft_consensus.cc:2454] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:50.383308 6305 raft_consensus.cc:2454] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:50.383512 6503 leader_election.cc:302] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34abd76ae3764e7f9641eaddaffdd5a5, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:50.383682 7347 raft_consensus.cc:2781] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.383735 7347 raft_consensus.cc:482] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.383759 7347 raft_consensus.cc:3037] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.384497 7347 raft_consensus.cc:504] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.384759 7347 leader_election.cc:288] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:50.385021 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b10f956b54ebaa82614f8a1f979d1" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:50.385102 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b10f956b54ebaa82614f8a1f979d1" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:50.385150 6699 raft_consensus.cc:3037] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.385213 6305 raft_consensus.cc:3037] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.385838 6699 raft_consensus.cc:2454] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:50.385915 6305 raft_consensus.cc:2454] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:50.386076 6503 leader_election.cc:302] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34abd76ae3764e7f9641eaddaffdd5a5, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:50.386211 7347 raft_consensus.cc:2781] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.386364 7347 raft_consensus.cc:686] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:50.386499 7347 consensus_queue.cc:237] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.387956 6164 catalog_manager.cc:5484] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:50.577854 7487 raft_consensus.cc:482] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.577991 7487 raft_consensus.cc:504] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:50.578403 7487 leader_election.cc:288] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:50.578744 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a934439594245d79ed5980ff9c03277" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:50.578814 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a934439594245d79ed5980ff9c03277" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:50.578876 6567 raft_consensus.cc:2454] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:50.578964 6699 raft_consensus.cc:2454] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:50.579187 6372 leader_election.cc:302] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:50.579347 7487 raft_consensus.cc:2781] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.579409 7487 raft_consensus.cc:482] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.579460 7487 raft_consensus.cc:3037] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.580264 7487 raft_consensus.cc:504] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:50.580562 7487 leader_election.cc:288] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Requested vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:50.580855 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a934439594245d79ed5980ff9c03277" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:50.580919 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a934439594245d79ed5980ff9c03277" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:50.580957 6567 raft_consensus.cc:3037] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.581036 6699 raft_consensus.cc:3037] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.581737 6699 raft_consensus.cc:2454] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:50.581737 6567 raft_consensus.cc:2454] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:50.582053 6372 leader_election.cc:302] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:50.582204 7487 raft_consensus.cc:2781] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.582286 7487 raft_consensus.cc:686] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Leader. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 07:59:50.582448 7487 consensus_queue.cc:237] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:50.583956 6164 catalog_manager.cc:5484] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:50.727425 7506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:50.727635 7506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:50.730535 7506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 07:59:50.791762 7347 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.795800 7503 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.797583 7355 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.802552 7355 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.842185 6699 raft_consensus.cc:1261] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.842530 7487 consensus_queue.cc:1035] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.849114 6567 raft_consensus.cc:1261] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.850095 7487 consensus_queue.cc:1035] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.436 ms
time per row: 0.0527178 ms
I20240503 07:59:50.873571 6164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:50.878310 6548 tablet_service.cc:1467] Processing CreateTablet for tablet a9023b7fb87a4adaa91995b883df8826 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.878405 6286 tablet_service.cc:1467] Processing CreateTablet for tablet a9023b7fb87a4adaa91995b883df8826 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.878597 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9023b7fb87a4adaa91995b883df8826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.878615 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9023b7fb87a4adaa91995b883df8826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.879124 6417 tablet_service.cc:1467] Processing CreateTablet for tablet 957ae2f17c4c41d28287d5490c54572e (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.879343 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957ae2f17c4c41d28287d5490c54572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.880776 6416 tablet_service.cc:1467] Processing CreateTablet for tablet a9023b7fb87a4adaa91995b883df8826 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.880999 6416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9023b7fb87a4adaa91995b883df8826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.883101 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 957ae2f17c4c41d28287d5490c54572e (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.883145 6285 tablet_service.cc:1467] Processing CreateTablet for tablet 64eb15b0505a4649a0043be00250c299 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.883311 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957ae2f17c4c41d28287d5490c54572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.883637 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64eb15b0505a4649a0043be00250c299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.883857 6678 tablet_service.cc:1467] Processing CreateTablet for tablet 957ae2f17c4c41d28287d5490c54572e (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.884092 6678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957ae2f17c4c41d28287d5490c54572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.885380 7568 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:50.886111 7570 tablet_bootstrap.cc:492] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:50.886154 7568 tablet_bootstrap.cc:654] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.886260 6679 tablet_service.cc:1467] Processing CreateTablet for tablet 64eb15b0505a4649a0043be00250c299 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.886449 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64eb15b0505a4649a0043be00250c299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.886756 7567 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:50.886844 7570 tablet_bootstrap.cc:654] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.887532 7567 tablet_bootstrap.cc:654] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.887715 7570 tablet_bootstrap.cc:492] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:50.887787 7570 ts_tablet_manager.cc:1367] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:50.888123 7569 tablet_bootstrap.cc:492] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:50.888145 7570 raft_consensus.cc:348] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.888234 7570 raft_consensus.cc:374] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.888263 7570 raft_consensus.cc:729] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.888377 7570 consensus_queue.cc:260] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.888628 7570 ts_tablet_manager.cc:1398] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:50.888778 7570 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:50.888825 7569 tablet_bootstrap.cc:654] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.889535 7570 tablet_bootstrap.cc:654] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.889503 6548 tablet_service.cc:1467] Processing CreateTablet for tablet 64eb15b0505a4649a0043be00250c299 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:50.889709 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64eb15b0505a4649a0043be00250c299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:50.891201 7570 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:50.891283 7570 ts_tablet_manager.cc:1367] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:50.891680 7570 raft_consensus.cc:348] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:50.891768 7570 raft_consensus.cc:374] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.891777 7568 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:50.891808 7570 raft_consensus.cc:729] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.891839 7568 ts_tablet_manager.cc:1367] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:50.891902 7570 consensus_queue.cc:260] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:50.892124 7570 ts_tablet_manager.cc:1398] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:50.892184 7568 raft_consensus.cc:348] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.892270 7568 raft_consensus.cc:374] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.892294 7568 raft_consensus.cc:729] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.892392 7568 consensus_queue.cc:260] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.892607 7568 ts_tablet_manager.cc:1398] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:50.892735 7568 tablet_bootstrap.cc:492] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:50.892863 7567 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:50.892936 7567 ts_tablet_manager.cc:1367] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:50.893482 7568 tablet_bootstrap.cc:654] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.893848 7567 raft_consensus.cc:348] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.893947 7567 raft_consensus.cc:374] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.893978 7567 raft_consensus.cc:729] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.894076 7567 consensus_queue.cc:260] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.894345 7567 ts_tablet_manager.cc:1398] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:50.894490 7567 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:50.895265 7567 tablet_bootstrap.cc:654] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.895274 7569 tablet_bootstrap.cc:492] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:50.895340 7569 ts_tablet_manager.cc:1367] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 07:59:50.895706 7569 raft_consensus.cc:348] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.895793 7569 raft_consensus.cc:374] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.895817 7569 raft_consensus.cc:729] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.895928 7569 consensus_queue.cc:260] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.896162 7569 ts_tablet_manager.cc:1398] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:50.896235 7567 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:50.896302 7567 ts_tablet_manager.cc:1367] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:50.896347 7569 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:50.896685 7567 raft_consensus.cc:348] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:50.896775 7567 raft_consensus.cc:374] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.896806 7567 raft_consensus.cc:729] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.896903 7567 consensus_queue.cc:260] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:50.897084 7569 tablet_bootstrap.cc:654] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.897167 7567 ts_tablet_manager.cc:1398] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:50.898677 7568 tablet_bootstrap.cc:492] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:50.898756 7568 ts_tablet_manager.cc:1367] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:50.899104 7568 raft_consensus.cc:348] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.899194 7568 raft_consensus.cc:374] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.899219 7568 raft_consensus.cc:729] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.899318 7568 consensus_queue.cc:260] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:50.899546 7568 ts_tablet_manager.cc:1398] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:50.899703 7568 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:50.900568 7568 tablet_bootstrap.cc:654] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:50.901469 7569 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:50.901540 7569 ts_tablet_manager.cc:1367] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:50.901894 7569 raft_consensus.cc:348] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.901991 7569 raft_consensus.cc:374] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.902024 7569 raft_consensus.cc:729] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.902123 7569 consensus_queue.cc:260] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.902367 7569 ts_tablet_manager.cc:1398] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:50.902940 7568 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:50.903013 7568 ts_tablet_manager.cc:1367] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:50.903331 7568 raft_consensus.cc:348] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:50.903416 7568 raft_consensus.cc:374] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:50.903442 7568 raft_consensus.cc:729] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:50.903535 7568 consensus_queue.cc:260] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:50.903757 7568 ts_tablet_manager.cc:1398] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:50.961166 7553 raft_consensus.cc:482] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.961280 7553 raft_consensus.cc:504] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.961678 7553 leader_election.cc:288] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:50.962000 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9023b7fb87a4adaa91995b883df8826" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:50.962026 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9023b7fb87a4adaa91995b883df8826" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:50.962121 6305 raft_consensus.cc:2454] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:50.962183 6437 raft_consensus.cc:2454] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:50.962407 6502 leader_election.cc:302] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:50.962613 7553 raft_consensus.cc:2781] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.962666 7553 raft_consensus.cc:482] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.962698 7553 raft_consensus.cc:3037] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.963689 7553 raft_consensus.cc:504] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.963986 7553 leader_election.cc:288] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:50.964232 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9023b7fb87a4adaa91995b883df8826" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:50.964262 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9023b7fb87a4adaa91995b883df8826" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:50.964332 6305 raft_consensus.cc:3037] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.964361 6437 raft_consensus.cc:3037] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.965030 6305 raft_consensus.cc:2454] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:50.965030 6437 raft_consensus.cc:2454] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:50.965293 6501 leader_election.cc:302] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:50.965444 7553 raft_consensus.cc:2781] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.965531 7553 raft_consensus.cc:686] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:50.965669 7553 consensus_queue.cc:237] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:50.967304 6164 catalog_manager.cc:5484] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:50.999742 7304 raft_consensus.cc:482] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.999850 7304 raft_consensus.cc:504] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.000231 7304 leader_election.cc:288] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:51.000589 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64eb15b0505a4649a0043be00250c299" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:51.000594 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64eb15b0505a4649a0043be00250c299" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:51.000721 6305 raft_consensus.cc:2454] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:51.000721 6567 raft_consensus.cc:2454] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:51.000965 6633 leader_election.cc:302] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:51.001155 7304 raft_consensus.cc:2781] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.001214 7304 raft_consensus.cc:482] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.001240 7304 raft_consensus.cc:3037] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.001997 7304 raft_consensus.cc:504] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.002287 7304 leader_election.cc:288] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Requested vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:51.002542 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64eb15b0505a4649a0043be00250c299" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:51.002640 6305 raft_consensus.cc:3037] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.002704 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64eb15b0505a4649a0043be00250c299" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:51.002820 6567 raft_consensus.cc:3037] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.003343 6305 raft_consensus.cc:2454] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:51.003481 6567 raft_consensus.cc:2454] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:51.003602 6633 leader_election.cc:302] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:51.003753 7304 raft_consensus.cc:2781] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.003880 7304 raft_consensus.cc:686] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Becoming Leader. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Running, Role: LEADER
I20240503 07:59:51.004010 7304 consensus_queue.cc:237] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.005455 6164 catalog_manager.cc:5484] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9976bc168264eaeba1d180fb38b5113 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.050202 7355 raft_consensus.cc:482] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.050305 7355 raft_consensus.cc:504] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.050690 7355 leader_election.cc:288] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:51.051046 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957ae2f17c4c41d28287d5490c54572e" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:51.051060 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957ae2f17c4c41d28287d5490c54572e" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:51.051168 6437 raft_consensus.cc:2454] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:51.051187 6699 raft_consensus.cc:2454] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:51.051435 6241 leader_election.cc:302] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:51.051621 7355 raft_consensus.cc:2781] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.051687 7355 raft_consensus.cc:482] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.051723 7355 raft_consensus.cc:3037] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.052450 7355 raft_consensus.cc:504] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.052731 7355 leader_election.cc:288] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:51.053037 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957ae2f17c4c41d28287d5490c54572e" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:51.053167 6437 raft_consensus.cc:3037] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.053160 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957ae2f17c4c41d28287d5490c54572e" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:51.053265 6699 raft_consensus.cc:3037] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.053861 6437 raft_consensus.cc:2454] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:51.054127 6699 raft_consensus.cc:2454] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:51.054152 6239 leader_election.cc:302] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:51.054388 7355 raft_consensus.cc:2781] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.054474 7355 raft_consensus.cc:686] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Becoming Leader. State: Replica: 253dd189c3c542248402cba222394f43, State: Running, Role: LEADER
I20240503 07:59:51.054621 7355 consensus_queue.cc:237] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.056164 6164 catalog_manager.cc:5484] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 253dd189c3c542248402cba222394f43 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:51.171200 7584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:51.171419 7584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:51.174361 7584 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 07:59:51.270699 6305 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.271099 7553 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.275003 6437 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.275422 6437 raft_consensus.cc:1261] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.275825 7355 consensus_queue.cc:1035] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.275944 6567 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.276392 7304 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.279511 7503 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.280064 6699 raft_consensus.cc:1261] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.280715 7355 consensus_queue.cc:1035] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.282752 6305 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.285480 7304 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 130.628 ms
time per row: 0.0653141 ms
I20240503 07:59:51.343597 6163 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:51.348263 6679 tablet_service.cc:1467] Processing CreateTablet for tablet d17b7dcf55364209bd26129488dd0184 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.348557 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b7dcf55364209bd26129488dd0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.349123 6548 tablet_service.cc:1467] Processing CreateTablet for tablet d17b7dcf55364209bd26129488dd0184 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.349371 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b7dcf55364209bd26129488dd0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.350289 7570 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:51.350477 6546 tablet_service.cc:1467] Processing CreateTablet for tablet 82f6c17706cc4a40827501ecb7c3ebce (DEFAULT_TABLE table=rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.350677 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f6c17706cc4a40827501ecb7c3ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.351130 7570 tablet_bootstrap.cc:654] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.351171 7567 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:51.351868 7567 tablet_bootstrap.cc:654] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.352945 6416 tablet_service.cc:1467] Processing CreateTablet for tablet d17b7dcf55364209bd26129488dd0184 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.353011 6415 tablet_service.cc:1467] Processing CreateTablet for tablet e03d7158b64f434a8871f009864a0b72 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.353024 6417 tablet_service.cc:1467] Processing CreateTablet for tablet 82f6c17706cc4a40827501ecb7c3ebce (DEFAULT_TABLE table=rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.353266 6416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b7dcf55364209bd26129488dd0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.353340 6679 tablet_service.cc:1467] Processing CreateTablet for tablet 82f6c17706cc4a40827501ecb7c3ebce (DEFAULT_TABLE table=rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.353538 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f6c17706cc4a40827501ecb7c3ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.355497 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03d7158b64f434a8871f009864a0b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.355590 7569 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:51.356343 7569 tablet_bootstrap.cc:654] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.357012 6285 tablet_service.cc:1467] Processing CreateTablet for tablet e03d7158b64f434a8871f009864a0b72 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.357308 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03d7158b64f434a8871f009864a0b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.357270 6547 tablet_service.cc:1467] Processing CreateTablet for tablet e03d7158b64f434a8871f009864a0b72 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.357462 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03d7158b64f434a8871f009864a0b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.359138 7568 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:51.359241 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f6c17706cc4a40827501ecb7c3ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.359777 7568 tablet_bootstrap.cc:654] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.360658 7568 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:51.360736 7568 ts_tablet_manager.cc:1367] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:51.361284 7568 raft_consensus.cc:348] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.361404 7568 raft_consensus.cc:374] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.361438 7568 raft_consensus.cc:729] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.361541 7568 consensus_queue.cc:260] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.361750 7567 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:51.361788 7568 ts_tablet_manager.cc:1398] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.361828 7567 ts_tablet_manager.cc:1367] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 07:59:51.362236 7567 raft_consensus.cc:348] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.362329 7567 raft_consensus.cc:374] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.362361 7567 raft_consensus.cc:729] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.362466 7567 consensus_queue.cc:260] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.362705 7567 ts_tablet_manager.cc:1398] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.362846 7567 tablet_bootstrap.cc:492] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:51.363595 7567 tablet_bootstrap.cc:654] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.364482 7567 tablet_bootstrap.cc:492] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:51.364567 7567 ts_tablet_manager.cc:1367] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:51.365085 7567 raft_consensus.cc:348] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.365180 7567 raft_consensus.cc:374] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.365216 7567 raft_consensus.cc:729] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.365332 7567 consensus_queue.cc:260] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.365664 7567 ts_tablet_manager.cc:1398] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.365816 7567 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:51.366598 7567 tablet_bootstrap.cc:654] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.367444 7567 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:51.367544 7567 ts_tablet_manager.cc:1367] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:51.367892 7567 raft_consensus.cc:348] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.367966 7567 raft_consensus.cc:374] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.367995 7567 raft_consensus.cc:729] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.368095 7567 consensus_queue.cc:260] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.368247 7569 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:51.368319 7569 ts_tablet_manager.cc:1367] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20240503 07:59:51.368407 7567 ts_tablet_manager.cc:1398] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.368661 7569 raft_consensus.cc:348] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.368762 7569 raft_consensus.cc:374] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.368793 7569 raft_consensus.cc:729] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.368911 7569 consensus_queue.cc:260] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.369197 7569 ts_tablet_manager.cc:1398] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:51.369349 7569 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:51.370082 7569 tablet_bootstrap.cc:654] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.371498 7570 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:51.371572 7569 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:51.371589 7570 ts_tablet_manager.cc:1367] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:51.371630 7569 ts_tablet_manager.cc:1367] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.371937 7570 raft_consensus.cc:348] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.371953 7569 raft_consensus.cc:348] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.372005 7570 raft_consensus.cc:374] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.372025 7570 raft_consensus.cc:729] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.372027 7569 raft_consensus.cc:374] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.372052 7569 raft_consensus.cc:729] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.372112 7570 consensus_queue.cc:260] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.372143 7569 consensus_queue.cc:260] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.372328 7570 ts_tablet_manager.cc:1398] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.372349 7569 ts_tablet_manager.cc:1398] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.372442 7570 tablet_bootstrap.cc:492] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:51.372475 7569 tablet_bootstrap.cc:492] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:51.373088 7570 tablet_bootstrap.cc:654] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.373373 7569 tablet_bootstrap.cc:654] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.374392 7570 tablet_bootstrap.cc:492] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:51.374471 7570 ts_tablet_manager.cc:1367] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.374855 7570 raft_consensus.cc:348] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.374935 7570 raft_consensus.cc:374] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.374959 7570 raft_consensus.cc:729] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.375049 7570 consensus_queue.cc:260] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.375245 7569 tablet_bootstrap.cc:492] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:51.375264 7570 ts_tablet_manager.cc:1398] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.375311 7569 ts_tablet_manager.cc:1367] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:51.375619 7569 raft_consensus.cc:348] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.375713 7569 raft_consensus.cc:374] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.375754 7569 raft_consensus.cc:729] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.375855 7569 consensus_queue.cc:260] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.376081 7569 ts_tablet_manager.cc:1398] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.444236 7565 raft_consensus.cc:482] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.444330 7565 raft_consensus.cc:504] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.444679 7565 leader_election.cc:288] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:51.444988 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:51.445093 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:51.445155 6699 raft_consensus.cc:2454] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:51.445231 6567 raft_consensus.cc:2454] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:51.445439 6372 leader_election.cc:302] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:51.445583 7565 raft_consensus.cc:2781] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.445636 7565 raft_consensus.cc:482] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.445672 7565 raft_consensus.cc:3037] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.446403 7565 raft_consensus.cc:504] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.446671 7565 leader_election.cc:288] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:51.446949 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:51.447077 6699 raft_consensus.cc:3037] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.447024 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:51.447154 6567 raft_consensus.cc:3037] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.447896 6699 raft_consensus.cc:2454] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:51.447952 6567 raft_consensus.cc:2454] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:51.448170 6372 leader_election.cc:302] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:51.448309 7565 raft_consensus.cc:2781] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.448462 7565 raft_consensus.cc:686] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Leader. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 07:59:51.448588 7565 consensus_queue.cc:237] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.449949 6161 catalog_manager.cc:5484] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.527727 7565 raft_consensus.cc:482] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.527828 7565 raft_consensus.cc:504] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.528209 7565 leader_election.cc:288] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:51.528510 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f6c17706cc4a40827501ecb7c3ebce" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:51.528551 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f6c17706cc4a40827501ecb7c3ebce" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:51.528640 6699 raft_consensus.cc:2454] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:51.528700 6567 raft_consensus.cc:2454] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:51.528964 6372 leader_election.cc:302] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:51.529132 7565 raft_consensus.cc:2781] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.529189 7565 raft_consensus.cc:482] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.529233 7565 raft_consensus.cc:3037] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.553755 7565 raft_consensus.cc:504] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.554103 7565 leader_election.cc:288] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:51.554461 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f6c17706cc4a40827501ecb7c3ebce" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:51.554514 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f6c17706cc4a40827501ecb7c3ebce" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:51.554575 6699 raft_consensus.cc:3037] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.554630 6567 raft_consensus.cc:3037] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.555320 6699 raft_consensus.cc:2454] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:51.555311 6567 raft_consensus.cc:2454] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:51.555572 6372 leader_election.cc:302] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:51.555739 7565 raft_consensus.cc:2781] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.555824 7565 raft_consensus.cc:686] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Leader. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 07:59:51.555972 7565 consensus_queue.cc:237] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.557468 6161 catalog_manager.cc:5484] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.578630 7534 raft_consensus.cc:482] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.578738 7534 raft_consensus.cc:504] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.579090 7534 leader_election.cc:288] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:51.579387 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d7158b64f434a8871f009864a0b72" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:51.579396 6436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d7158b64f434a8871f009864a0b72" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:51.579505 6567 raft_consensus.cc:2454] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:51.579510 6436 raft_consensus.cc:2454] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:51.579785 6239 leader_election.cc:302] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:51.579939 7534 raft_consensus.cc:2781] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.579994 7534 raft_consensus.cc:482] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.580034 7534 raft_consensus.cc:3037] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.580740 7534 raft_consensus.cc:504] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.581007 7534 leader_election.cc:288] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:51.581298 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d7158b64f434a8871f009864a0b72" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:51.581328 6436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e03d7158b64f434a8871f009864a0b72" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:51.581388 6567 raft_consensus.cc:3037] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.581415 6436 raft_consensus.cc:3037] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.582207 6567 raft_consensus.cc:2454] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:51.582213 6436 raft_consensus.cc:2454] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:51.582439 6239 leader_election.cc:302] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:51.582577 7534 raft_consensus.cc:2781] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.582661 7534 raft_consensus.cc:686] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Becoming Leader. State: Replica: 253dd189c3c542248402cba222394f43, State: Running, Role: LEADER
I20240503 07:59:51.582785 7534 consensus_queue.cc:237] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.584182 6161 catalog_manager.cc:5484] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 253dd189c3c542248402cba222394f43 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:51.741328 7635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:51.741564 7635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:51.744572 7635 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 07:59:51.815539 6699 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.815649 6567 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.815917 6566 raft_consensus.cc:1261] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.815953 6568 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.816082 7565 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.816272 7534 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.816450 7565 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.816701 7565 consensus_queue.cc:1035] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.817126 6699 raft_consensus.cc:1261] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.817484 7631 consensus_queue.cc:1035] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.824290 6436 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:51.826150 7355 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:51.831079 7543 mvcc.cc:205] Tried to move back new op lower bound from 7023506193673732096 to 7023506192724176896. Current Snapshot: MvccSnapshot[applied={T|T < 7023506193673732096}]
I20240503 07:59:51.839087 7531 mvcc.cc:205] Tried to move back new op lower bound from 7023506193672589312 to 7023506192614285312. Current Snapshot: MvccSnapshot[applied={T|T < 7023506193672589312}]
INSERT report
rows total: 2000
time total: 85.1707 ms
time per row: 0.0425854 ms
I20240503 07:59:51.871487 6163 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:51.876329 6417 tablet_service.cc:1467] Processing CreateTablet for tablet 551b0bab14bb4bb492885bada3e031a0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.876587 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551b0bab14bb4bb492885bada3e031a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.876827 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 94f828890165466787015f21507e6ba2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.877033 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f828890165466787015f21507e6ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.877324 6679 tablet_service.cc:1467] Processing CreateTablet for tablet 551b0bab14bb4bb492885bada3e031a0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.877542 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551b0bab14bb4bb492885bada3e031a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.877952 6285 tablet_service.cc:1467] Processing CreateTablet for tablet 551b0bab14bb4bb492885bada3e031a0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.878150 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551b0bab14bb4bb492885bada3e031a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.878649 7687 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:51.879196 6547 tablet_service.cc:1467] Processing CreateTablet for tablet 94f828890165466787015f21507e6ba2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.879348 6678 tablet_service.cc:1467] Processing CreateTablet for tablet 94f828890165466787015f21507e6ba2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.879413 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f828890165466787015f21507e6ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.879562 6678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f828890165466787015f21507e6ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.879570 7687 tablet_bootstrap.cc:654] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.879966 6677 tablet_service.cc:1467] Processing CreateTablet for tablet d74a03a296dc48e582faeddc7b978ccd (DEFAULT_TABLE table=rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.880158 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74a03a296dc48e582faeddc7b978ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.880595 6415 tablet_service.cc:1467] Processing CreateTablet for tablet d74a03a296dc48e582faeddc7b978ccd (DEFAULT_TABLE table=rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.880832 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74a03a296dc48e582faeddc7b978ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.881675 6285 tablet_service.cc:1467] Processing CreateTablet for tablet d74a03a296dc48e582faeddc7b978ccd (DEFAULT_TABLE table=rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:51.881883 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74a03a296dc48e582faeddc7b978ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:51.883522 7687 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:51.883631 7687 ts_tablet_manager.cc:1367] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:51.884052 7687 raft_consensus.cc:348] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.884153 7687 raft_consensus.cc:374] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.884191 7687 raft_consensus.cc:729] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.884303 7687 consensus_queue.cc:260] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.884580 7687 ts_tablet_manager.cc:1398] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.884799 7687 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:51.885210 7690 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:51.885685 7687 tablet_bootstrap.cc:654] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.886698 7687 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:51.886770 7690 tablet_bootstrap.cc:654] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.886775 7687 ts_tablet_manager.cc:1367] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:51.887128 7687 raft_consensus.cc:348] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.887207 7687 raft_consensus.cc:374] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.887233 7687 raft_consensus.cc:729] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.887332 7687 consensus_queue.cc:260] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.887405 7689 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:51.887550 7687 ts_tablet_manager.cc:1398] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.887816 7687 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:51.888295 7689 tablet_bootstrap.cc:654] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.888532 7687 tablet_bootstrap.cc:654] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.889336 7689 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:51.889420 7689 ts_tablet_manager.cc:1367] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.889454 7687 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:51.889523 7687 ts_tablet_manager.cc:1367] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.889813 7689 raft_consensus.cc:348] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.889886 7689 raft_consensus.cc:374] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.889906 7689 raft_consensus.cc:729] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.889870 7687 raft_consensus.cc:348] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.889952 7687 raft_consensus.cc:374] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.889981 7687 raft_consensus.cc:729] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.889997 7689 consensus_queue.cc:260] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.890060 7687 consensus_queue.cc:260] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.890241 7689 ts_tablet_manager.cc:1398] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:51.890280 7687 ts_tablet_manager.cc:1398] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.893589 7690 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:51.893671 7690 ts_tablet_manager.cc:1367] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:51.894050 7690 raft_consensus.cc:348] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.894168 7690 raft_consensus.cc:374] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.894199 7690 raft_consensus.cc:729] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.894306 7690 consensus_queue.cc:260] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.894557 7690 ts_tablet_manager.cc:1398] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.894698 7690 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:51.895321 7690 tablet_bootstrap.cc:654] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.896219 7690 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:51.896312 7690 ts_tablet_manager.cc:1367] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.896661 7690 raft_consensus.cc:348] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.896746 7690 raft_consensus.cc:374] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.896772 7690 raft_consensus.cc:729] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.896873 7690 consensus_queue.cc:260] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.897576 7690 ts_tablet_manager.cc:1398] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.910526 7696 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:51.911340 7696 tablet_bootstrap.cc:654] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.912142 7696 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:51.912207 7696 ts_tablet_manager.cc:1367] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.912518 7696 raft_consensus.cc:348] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.912597 7696 raft_consensus.cc:374] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.912621 7696 raft_consensus.cc:729] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.912711 7696 consensus_queue.cc:260] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.912915 7696 ts_tablet_manager.cc:1398] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.913089 7696 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:51.913734 7696 tablet_bootstrap.cc:654] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.914558 7696 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:51.914637 7696 ts_tablet_manager.cc:1367] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.915043 7696 raft_consensus.cc:348] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.915143 7696 raft_consensus.cc:374] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.915186 7696 raft_consensus.cc:729] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.915295 7696 consensus_queue.cc:260] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.915546 7696 ts_tablet_manager.cc:1398] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:51.915757 7696 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:51.916661 7696 tablet_bootstrap.cc:654] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:51.917641 7696 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:51.917721 7696 ts_tablet_manager.cc:1367] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:51.918198 7696 raft_consensus.cc:348] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.918272 7696 raft_consensus.cc:374] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:51.918298 7696 raft_consensus.cc:729] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:51.918403 7696 consensus_queue.cc:260] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:51.918648 7696 ts_tablet_manager.cc:1398] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:51.964515 7673 raft_consensus.cc:482] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.964614 7673 raft_consensus.cc:504] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.964938 7673 leader_election.cc:288] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:51.965196 6697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74a03a296dc48e582faeddc7b978ccd" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:51.965245 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74a03a296dc48e582faeddc7b978ccd" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:51.965323 6697 raft_consensus.cc:2454] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:51.965355 6306 raft_consensus.cc:2454] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:51.965574 6372 leader_election.cc:302] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:51.965736 7673 raft_consensus.cc:2781] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.965798 7673 raft_consensus.cc:482] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.965824 7673 raft_consensus.cc:3037] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.966066 7581 raft_consensus.cc:482] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:51.966136 7581 raft_consensus.cc:504] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.966437 7581 leader_election.cc:288] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:51.966719 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551b0bab14bb4bb492885bada3e031a0" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:51.966725 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551b0bab14bb4bb492885bada3e031a0" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:51.966714 7673 raft_consensus.cc:504] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.966821 6306 raft_consensus.cc:2454] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:51.966851 6437 raft_consensus.cc:2454] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:51.967049 7673 leader_election.cc:288] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:51.967046 6633 leader_election.cc:302] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:51.967228 7581 raft_consensus.cc:2781] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:51.967288 7581 raft_consensus.cc:482] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:51.967332 7581 raft_consensus.cc:3037] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.967334 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74a03a296dc48e582faeddc7b978ccd" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:51.967399 6697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74a03a296dc48e582faeddc7b978ccd" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:51.967417 6306 raft_consensus.cc:3037] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.967487 6697 raft_consensus.cc:3037] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.968171 6306 raft_consensus.cc:2454] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:51.968235 7581 raft_consensus.cc:504] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.968286 6697 raft_consensus.cc:2454] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:51.968472 6371 leader_election.cc:302] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:51.968567 7581 leader_election.cc:288] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:51.968772 7673 raft_consensus.cc:2781] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.968780 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551b0bab14bb4bb492885bada3e031a0" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:51.968858 7673 raft_consensus.cc:686] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Leader. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 07:59:51.968849 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551b0bab14bb4bb492885bada3e031a0" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:51.968904 6437 raft_consensus.cc:3037] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.968915 6306 raft_consensus.cc:3037] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:51.968973 7673 consensus_queue.cc:237] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.969691 6306 raft_consensus.cc:2454] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:51.969851 6437 raft_consensus.cc:2454] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:51.969996 6633 leader_election.cc:302] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:51.970366 7581 raft_consensus.cc:2781] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:51.970561 7581 raft_consensus.cc:686] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Becoming Leader. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Running, Role: LEADER
I20240503 07:59:51.970682 7581 consensus_queue.cc:237] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:51.971011 6161 catalog_manager.cc:5484] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.973435 6163 catalog_manager.cc:5484] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9976bc168264eaeba1d180fb38b5113 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.102983 7534 raft_consensus.cc:482] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.103127 7534 raft_consensus.cc:504] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.103595 7534 leader_election.cc:288] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:52.104000 6697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f828890165466787015f21507e6ba2" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:52.104023 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f828890165466787015f21507e6ba2" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:52.104125 6697 raft_consensus.cc:2454] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:52.104173 6568 raft_consensus.cc:2454] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:52.104422 6241 leader_election.cc:302] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:52.104606 7534 raft_consensus.cc:2781] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.104657 7534 raft_consensus.cc:482] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.104681 7534 raft_consensus.cc:3037] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.105520 7534 raft_consensus.cc:504] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.105790 7534 leader_election.cc:288] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:52.106086 6697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f828890165466787015f21507e6ba2" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:52.106161 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f828890165466787015f21507e6ba2" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:52.106189 6697 raft_consensus.cc:3037] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.106258 6568 raft_consensus.cc:3037] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.106894 6697 raft_consensus.cc:2454] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:52.106933 6568 raft_consensus.cc:2454] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:52.107162 6241 leader_election.cc:302] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:52.107316 7355 raft_consensus.cc:2781] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.107389 7355 raft_consensus.cc:686] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Becoming Leader. State: Replica: 253dd189c3c542248402cba222394f43, State: Running, Role: LEADER
I20240503 07:59:52.107527 7355 consensus_queue.cc:237] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.109238 6163 catalog_manager.cc:5484] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 253dd189c3c542248402cba222394f43 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:52.270066 7704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:52.270253 7704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:52.273249 7704 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 07:59:52.359558 6568 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:52.359645 6697 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:52.360037 7355 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:52.360198 7534 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:52.361049 6699 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:52.361519 7674 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:52.362031 6306 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:52.363000 7674 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:52.364109 6305 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:52.364478 6437 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:52.364784 7581 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:52.364908 7701 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:52.381795 7677 mvcc.cc:205] Tried to move back new op lower bound from 7023506195920408576 to 7023506194312802304. Current Snapshot: MvccSnapshot[applied={T|T < 7023506195920408576}]
INSERT report
rows total: 2000
time total: 127.44 ms
time per row: 0.06372 ms
I20240503 07:59:52.438320 6159 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:52.443133 6415 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8a5e7a21684a4782d43c5a1131f652 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.443236 6285 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8a5e7a21684a4782d43c5a1131f652 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.443421 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8a5e7a21684a4782d43c5a1131f652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.443445 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8a5e7a21684a4782d43c5a1131f652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.443722 6547 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8a5e7a21684a4782d43c5a1131f652 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.443934 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8a5e7a21684a4782d43c5a1131f652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.445503 7761 tablet_bootstrap.cc:492] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:52.445683 6286 tablet_service.cc:1467] Processing CreateTablet for tablet d7012d8274214cf58c70d52062474b3a (DEFAULT_TABLE table=rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.445891 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7012d8274214cf58c70d52062474b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.446117 6678 tablet_service.cc:1467] Processing CreateTablet for tablet 8625e8b653624ea6acf6e04c3fd352a4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.446336 7761 tablet_bootstrap.cc:654] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.446367 6678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8625e8b653624ea6acf6e04c3fd352a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.446506 7762 tablet_bootstrap.cc:492] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:52.447358 7762 tablet_bootstrap.cc:654] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.448119 7763 tablet_bootstrap.cc:492] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:52.448187 6415 tablet_service.cc:1467] Processing CreateTablet for tablet 8625e8b653624ea6acf6e04c3fd352a4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.448187 6677 tablet_service.cc:1467] Processing CreateTablet for tablet d7012d8274214cf58c70d52062474b3a (DEFAULT_TABLE table=rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.448392 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8625e8b653624ea6acf6e04c3fd352a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.448391 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7012d8274214cf58c70d52062474b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.448949 7762 tablet_bootstrap.cc:492] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:52.448984 7763 tablet_bootstrap.cc:654] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.449020 7762 ts_tablet_manager.cc:1367] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:52.449436 7762 raft_consensus.cc:348] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.449535 7762 raft_consensus.cc:374] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.449566 7762 raft_consensus.cc:729] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.449702 7762 consensus_queue.cc:260] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.450021 7763 tablet_bootstrap.cc:492] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:52.450088 7763 ts_tablet_manager.cc:1367] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:52.450423 6547 tablet_service.cc:1467] Processing CreateTablet for tablet 8625e8b653624ea6acf6e04c3fd352a4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.450469 7763 raft_consensus.cc:348] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.450567 7763 raft_consensus.cc:374] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.450505 6546 tablet_service.cc:1467] Processing CreateTablet for tablet d7012d8274214cf58c70d52062474b3a (DEFAULT_TABLE table=rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:52.450596 7763 raft_consensus.cc:729] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.450632 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8625e8b653624ea6acf6e04c3fd352a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.450686 7763 consensus_queue.cc:260] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.450738 7766 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:52.450737 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7012d8274214cf58c70d52062474b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:52.450942 7763 ts_tablet_manager.cc:1398] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.451089 7763 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:52.451489 7766 tablet_bootstrap.cc:654] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.451985 7763 tablet_bootstrap.cc:654] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.452397 7766 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:52.452466 7766 ts_tablet_manager.cc:1367] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:52.452852 7766 raft_consensus.cc:348] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.452934 7766 raft_consensus.cc:374] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.452965 7766 raft_consensus.cc:729] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.453087 7766 consensus_queue.cc:260] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.453150 7762 ts_tablet_manager.cc:1398] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:52.453294 7762 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:52.453344 7766 ts_tablet_manager.cc:1398] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.453490 7766 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:52.453831 7763 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:52.453905 7763 ts_tablet_manager.cc:1367] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:52.454267 7762 tablet_bootstrap.cc:654] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.454277 7766 tablet_bootstrap.cc:654] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.454391 7763 raft_consensus.cc:348] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.454492 7763 raft_consensus.cc:374] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.454540 7763 raft_consensus.cc:729] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.454670 7763 consensus_queue.cc:260] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.454988 7763 ts_tablet_manager.cc:1398] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:52.455361 7761 tablet_bootstrap.cc:492] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:52.455463 7762 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:52.455480 7761 ts_tablet_manager.cc:1367] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:52.455549 7762 ts_tablet_manager.cc:1367] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:52.455955 7762 raft_consensus.cc:348] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.456055 7762 raft_consensus.cc:374] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.456094 7762 raft_consensus.cc:729] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.456203 7762 consensus_queue.cc:260] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.456451 7762 ts_tablet_manager.cc:1398] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:52.456400 7761 raft_consensus.cc:348] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.457007 7761 raft_consensus.cc:374] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.457121 7761 raft_consensus.cc:729] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.457235 7766 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:52.457311 7766 ts_tablet_manager.cc:1367] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:52.457322 7761 consensus_queue.cc:260] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.457661 7761 ts_tablet_manager.cc:1398] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:52.457922 7761 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:52.458808 7761 tablet_bootstrap.cc:654] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.459332 7766 raft_consensus.cc:348] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.459422 7766 raft_consensus.cc:374] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.459450 7766 raft_consensus.cc:729] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.459578 7766 consensus_queue.cc:260] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.459884 7766 ts_tablet_manager.cc:1398] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:52.460302 7761 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:52.460572 7761 ts_tablet_manager.cc:1367] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:52.461131 7761 raft_consensus.cc:348] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.461217 7761 raft_consensus.cc:374] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.461247 7761 raft_consensus.cc:729] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.461351 7761 consensus_queue.cc:260] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.461581 7761 ts_tablet_manager.cc:1398] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:52.461704 7761 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:52.462388 7761 tablet_bootstrap.cc:654] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:52.463657 7761 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:52.463744 7761 ts_tablet_manager.cc:1367] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:52.464151 7761 raft_consensus.cc:348] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.464226 7761 raft_consensus.cc:374] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:52.464246 7761 raft_consensus.cc:729] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:52.464332 7761 consensus_queue.cc:260] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.464560 7761 ts_tablet_manager.cc:1398] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:52.485991 7503 raft_consensus.cc:482] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.486079 7503 raft_consensus.cc:504] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.486460 7503 leader_election.cc:288] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:52.486768 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8a5e7a21684a4782d43c5a1131f652" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:52.486781 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8a5e7a21684a4782d43c5a1131f652" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:52.486886 6305 raft_consensus.cc:2454] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:52.486893 6437 raft_consensus.cc:2454] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:52.487143 6502 leader_election.cc:302] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:52.487289 7503 raft_consensus.cc:2781] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.487339 7503 raft_consensus.cc:482] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.487365 7503 raft_consensus.cc:3037] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.488091 7503 raft_consensus.cc:504] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.488401 7503 leader_election.cc:288] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:52.488662 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8a5e7a21684a4782d43c5a1131f652" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:52.488703 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8a5e7a21684a4782d43c5a1131f652" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:52.488795 6437 raft_consensus.cc:3037] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.488814 6305 raft_consensus.cc:3037] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.489581 6437 raft_consensus.cc:2454] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:52.489666 6305 raft_consensus.cc:2454] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:52.489811 6501 leader_election.cc:302] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:52.489948 7503 raft_consensus.cc:2781] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.490128 7503 raft_consensus.cc:686] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:52.490252 7503 consensus_queue.cc:237] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.491641 6162 catalog_manager.cc:5484] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.567755 7503 raft_consensus.cc:482] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.567875 7503 raft_consensus.cc:504] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.568269 7503 leader_election.cc:288] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:52.568599 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:52.568598 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:52.568719 6437 raft_consensus.cc:2454] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:52.568730 6699 raft_consensus.cc:2454] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:52.569010 6501 leader_election.cc:302] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:52.569180 7503 raft_consensus.cc:2781] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.569231 7503 raft_consensus.cc:482] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.569257 7503 raft_consensus.cc:3037] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.569979 7503 raft_consensus.cc:504] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.570242 7503 leader_election.cc:288] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:52.570510 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:52.570540 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:52.570626 6437 raft_consensus.cc:3037] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.570611 6699 raft_consensus.cc:3037] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.571311 6437 raft_consensus.cc:2454] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:52.571486 6699 raft_consensus.cc:2454] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:52.571564 6501 leader_election.cc:302] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:52.571823 7503 raft_consensus.cc:2781] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.571916 7503 raft_consensus.cc:686] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:52.572181 7503 consensus_queue.cc:237] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:52.573657 6162 catalog_manager.cc:5484] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.806435 7581 raft_consensus.cc:482] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:52.806558 7581 raft_consensus.cc:504] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.806950 7581 leader_election.cc:288] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:52.807272 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7012d8274214cf58c70d52062474b3a" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:52.807286 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7012d8274214cf58c70d52062474b3a" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:52.807416 6305 raft_consensus.cc:2454] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:52.807413 6568 raft_consensus.cc:2454] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:52.807688 6633 leader_election.cc:302] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:52.807871 7581 raft_consensus.cc:2781] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:52.807929 7581 raft_consensus.cc:482] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:52.807965 7581 raft_consensus.cc:3037] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.808735 7581 raft_consensus.cc:504] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.809007 7581 leader_election.cc:288] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Requested vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:52.809314 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7012d8274214cf58c70d52062474b3a" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:52.809325 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7012d8274214cf58c70d52062474b3a" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:52.809422 6305 raft_consensus.cc:3037] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.809423 6568 raft_consensus.cc:3037] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:52.810324 6305 raft_consensus.cc:2454] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:52.810568 6633 leader_election.cc:302] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:52.810720 7581 raft_consensus.cc:2781] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:52.810737 6568 raft_consensus.cc:2454] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:52.810806 7581 raft_consensus.cc:686] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Becoming Leader. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Running, Role: LEADER
I20240503 07:59:52.810962 7581 consensus_queue.cc:237] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:52.812561 6162 catalog_manager.cc:5484] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9976bc168264eaeba1d180fb38b5113 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:52.993857 7778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:52.994084 7778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:52.997083 7778 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:53.005131 7503 consensus_queue.cc:1035] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.005450 7503 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.009416 7792 consensus_queue.cc:1035] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.012280 7503 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240503 07:59:53.074123 6305 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.074664 7701 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.075523 6568 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.079823 7701 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.107594 7791 mvcc.cc:205] Tried to move back new op lower bound from 7023506198827507712 to 7023506197754728448. Current Snapshot: MvccSnapshot[applied={T|T < 7023506198827507712 or (T in {7023506198827507712})}]
INSERT report
rows total: 2000
time total: 90.4743 ms
time per row: 0.0452371 ms
I20240503 07:59:53.131793 6163 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:53.136595 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1dab998bb84cc9880252383cb1e982 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.136847 6677 tablet_service.cc:1467] Processing CreateTablet for tablet 387f6b1606b94e11897372c0557870e4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.136837 6546 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1dab998bb84cc9880252383cb1e982 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.136906 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1dab998bb84cc9880252383cb1e982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.137117 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387f6b1606b94e11897372c0557870e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.137157 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1dab998bb84cc9880252383cb1e982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.137221 6415 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1dab998bb84cc9880252383cb1e982 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.137439 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1dab998bb84cc9880252383cb1e982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.139246 7831 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:53.139904 6415 tablet_service.cc:1467] Processing CreateTablet for tablet 387f6b1606b94e11897372c0557870e4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.140117 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387f6b1606b94e11897372c0557870e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.139926 6417 tablet_service.cc:1467] Processing CreateTablet for tablet 8efd24415e484e2a9af88b3aef6f78cc (DEFAULT_TABLE table=rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.140259 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efd24415e484e2a9af88b3aef6f78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.140280 7832 tablet_bootstrap.cc:492] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:53.140352 7831 tablet_bootstrap.cc:654] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.140529 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 387f6b1606b94e11897372c0557870e4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.140985 7833 tablet_bootstrap.cc:492] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:53.141000 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387f6b1606b94e11897372c0557870e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.141180 7832 tablet_bootstrap.cc:654] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.141866 7833 tablet_bootstrap.cc:654] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.142225 7832 tablet_bootstrap.cc:492] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:53.142309 7832 ts_tablet_manager.cc:1367] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:53.142755 7832 raft_consensus.cc:348] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.142865 7832 raft_consensus.cc:374] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.142901 7832 raft_consensus.cc:729] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.143015 7832 consensus_queue.cc:260] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.143173 6677 tablet_service.cc:1467] Processing CreateTablet for tablet 8efd24415e484e2a9af88b3aef6f78cc (DEFAULT_TABLE table=rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.143266 7832 ts_tablet_manager.cc:1398] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:53.143370 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efd24415e484e2a9af88b3aef6f78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.143582 6546 tablet_service.cc:1467] Processing CreateTablet for tablet 8efd24415e484e2a9af88b3aef6f78cc (DEFAULT_TABLE table=rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.143770 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efd24415e484e2a9af88b3aef6f78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.143312 7830 tablet_bootstrap.cc:492] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:53.145709 7831 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:53.145789 7831 ts_tablet_manager.cc:1367] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:53.145843 7832 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:53.146246 7830 tablet_bootstrap.cc:654] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.146219 7831 raft_consensus.cc:348] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:53.146308 7831 raft_consensus.cc:374] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.146337 7831 raft_consensus.cc:729] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.146448 7831 consensus_queue.cc:260] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:53.146615 7832 tablet_bootstrap.cc:654] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.146701 7831 ts_tablet_manager.cc:1398] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:53.147086 7831 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:53.147135 7833 tablet_bootstrap.cc:492] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:53.147203 7833 ts_tablet_manager.cc:1367] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:53.147588 7830 tablet_bootstrap.cc:492] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:53.147661 7830 ts_tablet_manager.cc:1367] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:53.147616 7833 raft_consensus.cc:348] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.147707 7833 raft_consensus.cc:374] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.147738 7833 raft_consensus.cc:729] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.147850 7833 consensus_queue.cc:260] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.147907 7831 tablet_bootstrap.cc:654] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.148101 7833 ts_tablet_manager.cc:1398] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:53.148072 7830 raft_consensus.cc:348] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.148165 7830 raft_consensus.cc:374] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.148198 7830 raft_consensus.cc:729] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.148332 7830 consensus_queue.cc:260] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.148854 7831 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:53.148923 7831 ts_tablet_manager.cc:1367] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:53.149233 7830 ts_tablet_manager.cc:1398] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:53.149327 7833 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:53.149319 7831 raft_consensus.cc:348] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:53.149392 7830 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:53.149406 7831 raft_consensus.cc:374] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.149435 7831 raft_consensus.cc:729] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.149547 7831 consensus_queue.cc:260] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:53.149823 7831 ts_tablet_manager.cc:1398] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:53.150090 7833 tablet_bootstrap.cc:654] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.150291 7830 tablet_bootstrap.cc:654] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.151302 7832 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:53.151381 7832 ts_tablet_manager.cc:1367] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:53.151789 7832 raft_consensus.cc:348] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:53.151891 7832 raft_consensus.cc:374] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.151924 7832 raft_consensus.cc:729] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.152128 7832 consensus_queue.cc:260] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:53.152458 7832 ts_tablet_manager.cc:1398] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:53.153329 7830 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:53.153399 7830 ts_tablet_manager.cc:1367] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:53.153539 7833 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:53.153625 7833 ts_tablet_manager.cc:1367] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:53.153959 7833 raft_consensus.cc:348] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:53.154057 7833 raft_consensus.cc:374] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.154096 7833 raft_consensus.cc:729] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.154204 7833 consensus_queue.cc:260] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:53.154192 7830 raft_consensus.cc:348] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:53.154290 7830 raft_consensus.cc:374] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.154325 7830 raft_consensus.cc:729] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.154430 7830 consensus_queue.cc:260] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:53.154465 7833 ts_tablet_manager.cc:1398] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.154628 7833 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:53.154655 7830 ts_tablet_manager.cc:1398] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.155371 7833 tablet_bootstrap.cc:654] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.156739 7833 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:53.156808 7833 ts_tablet_manager.cc:1367] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:53.157172 7833 raft_consensus.cc:348] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:53.157269 7833 raft_consensus.cc:374] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.157306 7833 raft_consensus.cc:729] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.157429 7833 consensus_queue.cc:260] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:53.157683 7833 ts_tablet_manager.cc:1398] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.317494 7798 raft_consensus.cc:482] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:53.317595 7798 raft_consensus.cc:504] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:53.317958 7798 leader_election.cc:288] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:53.318248 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387f6b1606b94e11897372c0557870e4" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:53.318253 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387f6b1606b94e11897372c0557870e4" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:53.318358 6437 raft_consensus.cc:2454] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:53.318384 6699 raft_consensus.cc:2454] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:53.318630 6239 leader_election.cc:302] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:53.318827 7798 raft_consensus.cc:2781] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:53.318884 7798 raft_consensus.cc:482] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:53.318921 7798 raft_consensus.cc:3037] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.319707 7798 raft_consensus.cc:504] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:53.319988 7798 leader_election.cc:288] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:53.320233 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387f6b1606b94e11897372c0557870e4" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:53.320250 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387f6b1606b94e11897372c0557870e4" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:53.320333 6699 raft_consensus.cc:3037] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.320343 6437 raft_consensus.cc:3037] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.320996 6699 raft_consensus.cc:2454] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:53.321008 6437 raft_consensus.cc:2454] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:53.321260 6241 leader_election.cc:302] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:53.321385 7798 raft_consensus.cc:2781] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:53.321482 7798 raft_consensus.cc:686] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Becoming Leader. State: Replica: 253dd189c3c542248402cba222394f43, State: Running, Role: LEADER
I20240503 07:59:53.321614 7798 consensus_queue.cc:237] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } }
I20240503 07:59:53.323228 6163 catalog_manager.cc:5484] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 253dd189c3c542248402cba222394f43 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:53.372572 7829 raft_consensus.cc:482] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:53.372699 7829 raft_consensus.cc:504] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:53.373126 7829 leader_election.cc:288] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:53.373474 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efd24415e484e2a9af88b3aef6f78cc" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:53.373476 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efd24415e484e2a9af88b3aef6f78cc" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:53.373605 6437 raft_consensus.cc:2454] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:53.373611 6568 raft_consensus.cc:2454] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 0.
I20240503 07:59:53.373890 6634 leader_election.cc:302] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34abd76ae3764e7f9641eaddaffdd5a5, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:53.374058 7829 raft_consensus.cc:2781] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:53.374114 7829 raft_consensus.cc:482] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:53.374141 7829 raft_consensus.cc:3037] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.374902 7829 raft_consensus.cc:504] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:53.375181 7829 leader_election.cc:288] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Requested vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:53.375469 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efd24415e484e2a9af88b3aef6f78cc" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:53.375566 6437 raft_consensus.cc:3037] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.376235 6437 raft_consensus.cc:2454] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:53.376435 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efd24415e484e2a9af88b3aef6f78cc" candidate_uuid: "e9976bc168264eaeba1d180fb38b5113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:53.376477 6632 leader_election.cc:302] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:53.376541 6568 raft_consensus.cc:3037] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.376612 7829 raft_consensus.cc:2781] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:53.376688 7829 raft_consensus.cc:686] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Becoming Leader. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Running, Role: LEADER
I20240503 07:59:53.376822 7829 consensus_queue.cc:237] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } }
I20240503 07:59:53.377481 6568 raft_consensus.cc:2454] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9976bc168264eaeba1d180fb38b5113 in term 1.
I20240503 07:59:53.378535 6163 catalog_manager.cc:5484] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9976bc168264eaeba1d180fb38b5113 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:53.418973 7674 raft_consensus.cc:482] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:53.419083 7674 raft_consensus.cc:504] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.419440 7674 leader_election.cc:288] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:53.419770 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1dab998bb84cc9880252383cb1e982" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:53.419829 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1dab998bb84cc9880252383cb1e982" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:53.419900 6305 raft_consensus.cc:2454] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:53.420150 6568 raft_consensus.cc:2454] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 0.
I20240503 07:59:53.420189 6371 leader_election.cc:302] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:53.420342 7674 raft_consensus.cc:2781] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:53.420408 7674 raft_consensus.cc:482] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:53.420444 7674 raft_consensus.cc:3037] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.421285 7674 raft_consensus.cc:504] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.421561 7674 leader_election.cc:288] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Requested vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:53.421862 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1dab998bb84cc9880252383cb1e982" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:53.421882 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1dab998bb84cc9880252383cb1e982" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:53.421963 6305 raft_consensus.cc:3037] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.421969 6568 raft_consensus.cc:3037] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.422613 6305 raft_consensus.cc:2454] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:53.422798 6568 raft_consensus.cc:2454] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 1.
I20240503 07:59:53.422894 6371 leader_election.cc:302] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43; no voters:
I20240503 07:59:53.423045 7674 raft_consensus.cc:2781] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:53.423203 7674 raft_consensus.cc:686] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Leader. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 07:59:53.423336 7674 consensus_queue.cc:237] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.424934 6163 catalog_manager.cc:5484] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:53.529482 7847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:53.529703 7847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:53.532681 7847 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 07:59:53.601243 6437 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.601644 7829 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.603227 6568 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.608142 7829 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.608412 6568 raft_consensus.cc:1261] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.611667 6437 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.612350 6699 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.612754 7798 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.613154 7884 consensus_queue.cc:1035] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.614436 7355 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.617709 6305 raft_consensus.cc:1261] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.626937 7884 consensus_queue.cc:1035] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 87.5891 ms
time per row: 0.0437945 ms
I20240503 07:59:53.670677 6160 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:53.675276 6546 tablet_service.cc:1467] Processing CreateTablet for tablet 7d608390b027432f8a3f89130c051759 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.675326 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 7d608390b027432f8a3f89130c051759 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.675375 6677 tablet_service.cc:1467] Processing CreateTablet for tablet 7d608390b027432f8a3f89130c051759 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.675556 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d608390b027432f8a3f89130c051759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.675582 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d608390b027432f8a3f89130c051759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.675582 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d608390b027432f8a3f89130c051759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.676069 6415 tablet_service.cc:1467] Processing CreateTablet for tablet 67b9602786f8422d962885dda25c6c85 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.676290 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b9602786f8422d962885dda25c6c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.678310 7899 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:53.678468 6547 tablet_service.cc:1467] Processing CreateTablet for tablet 67b9602786f8422d962885dda25c6c85 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.678675 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b9602786f8422d962885dda25c6c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.679047 6677 tablet_service.cc:1467] Processing CreateTablet for tablet aa0b29181066483386cab4c6f921f8f0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.679250 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa0b29181066483386cab4c6f921f8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.679352 7899 tablet_bootstrap.cc:654] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.680096 7900 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb: Bootstrap starting.
I20240503 07:59:53.680652 6546 tablet_service.cc:1467] Processing CreateTablet for tablet aa0b29181066483386cab4c6f921f8f0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.680866 7900 tablet_bootstrap.cc:654] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.680866 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa0b29181066483386cab4c6f921f8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.681793 7899 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:53.681866 7899 ts_tablet_manager.cc:1367] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:53.682245 6285 tablet_service.cc:1467] Processing CreateTablet for tablet aa0b29181066483386cab4c6f921f8f0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.682264 7899 raft_consensus.cc:348] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:53.682354 7899 raft_consensus.cc:374] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.682379 7899 raft_consensus.cc:729] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.682458 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa0b29181066483386cab4c6f921f8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.682476 7899 consensus_queue.cc:260] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:53.682694 7899 ts_tablet_manager.cc:1398] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.682998 6284 tablet_service.cc:1467] Processing CreateTablet for tablet 67b9602786f8422d962885dda25c6c85 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:53.683192 6284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b9602786f8422d962885dda25c6c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:53.684592 7903 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:53.685093 7900 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb: No bootstrap required, opened a new log
I20240503 07:59:53.685170 7900 ts_tablet_manager.cc:1367] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:53.685603 7903 tablet_bootstrap.cc:654] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.685582 7900 raft_consensus.cc:348] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.685686 7900 raft_consensus.cc:374] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.685717 7900 raft_consensus.cc:729] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.685832 7900 consensus_queue.cc:260] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.686491 7903 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:53.686556 7903 ts_tablet_manager.cc:1367] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:53.686579 7900 ts_tablet_manager.cc:1398] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.686801 7898 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:53.686946 7903 raft_consensus.cc:348] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:53.687036 7903 raft_consensus.cc:374] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.687064 7903 raft_consensus.cc:729] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.687166 7903 consensus_queue.cc:260] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:53.687405 7903 ts_tablet_manager.cc:1398] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.687549 7903 tablet_bootstrap.cc:492] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:53.687794 7898 tablet_bootstrap.cc:654] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.688165 7899 tablet_bootstrap.cc:492] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113: Bootstrap starting.
I20240503 07:59:53.688315 7903 tablet_bootstrap.cc:654] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.688995 7899 tablet_bootstrap.cc:654] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.689638 7903 tablet_bootstrap.cc:492] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:53.689705 7903 ts_tablet_manager.cc:1367] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:53.690095 7903 raft_consensus.cc:348] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.690248 7903 raft_consensus.cc:374] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.690306 7903 raft_consensus.cc:729] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.690418 7903 consensus_queue.cc:260] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.690692 7903 ts_tablet_manager.cc:1398] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.690836 7903 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43: Bootstrap starting.
I20240503 07:59:53.691347 7899 tablet_bootstrap.cc:492] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113: No bootstrap required, opened a new log
I20240503 07:59:53.691416 7899 ts_tablet_manager.cc:1367] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:53.691905 7899 raft_consensus.cc:348] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.691995 7899 raft_consensus.cc:374] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.692023 7899 raft_consensus.cc:729] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9976bc168264eaeba1d180fb38b5113, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.692135 7899 consensus_queue.cc:260] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.692350 7899 ts_tablet_manager.cc:1398] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.692399 7903 tablet_bootstrap.cc:654] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.693712 7903 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43: No bootstrap required, opened a new log
I20240503 07:59:53.693792 7903 ts_tablet_manager.cc:1367] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:53.693928 7898 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:53.693997 7898 ts_tablet_manager.cc:1367] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:53.694160 7903 raft_consensus.cc:348] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.694245 7903 raft_consensus.cc:374] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.694274 7903 raft_consensus.cc:729] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.694386 7903 consensus_queue.cc:260] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.694639 7903 ts_tablet_manager.cc:1398] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.694983 7898 raft_consensus.cc:348] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:53.695087 7898 raft_consensus.cc:374] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.695119 7898 raft_consensus.cc:729] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.695222 7898 consensus_queue.cc:260] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:53.695525 7898 ts_tablet_manager.cc:1398] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:53.695766 7898 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:53.697013 7898 tablet_bootstrap.cc:654] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.697924 7898 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:53.697993 7898 ts_tablet_manager.cc:1367] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:53.698321 7898 raft_consensus.cc:348] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.698398 7898 raft_consensus.cc:374] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.698418 7898 raft_consensus.cc:729] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.698504 7898 consensus_queue.cc:260] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.698717 7898 ts_tablet_manager.cc:1398] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:53.698876 7898 tablet_bootstrap.cc:492] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5: Bootstrap starting.
I20240503 07:59:53.699590 7898 tablet_bootstrap.cc:654] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:53.700450 7898 tablet_bootstrap.cc:492] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5: No bootstrap required, opened a new log
I20240503 07:59:53.700513 7898 ts_tablet_manager.cc:1367] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:53.700814 7898 raft_consensus.cc:348] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.700886 7898 raft_consensus.cc:374] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:53.700913 7898 raft_consensus.cc:729] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Initialized, Role: FOLLOWER
I20240503 07:59:53.701045 7898 consensus_queue.cc:260] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.701296 7898 ts_tablet_manager.cc:1398] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:53.712772 7792 raft_consensus.cc:482] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:53.712857 7792 raft_consensus.cc:504] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.713192 7792 leader_election.cc:288] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:53.713413 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9602786f8422d962885dda25c6c85" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:53.713517 6305 raft_consensus.cc:2454] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:53.713498 6436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9602786f8422d962885dda25c6c85" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" is_pre_election: true
I20240503 07:59:53.713610 6436 raft_consensus.cc:2454] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:53.713742 6502 leader_election.cc:302] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:53.713878 7792 raft_consensus.cc:2781] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:53.713936 7792 raft_consensus.cc:482] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:53.713964 7792 raft_consensus.cc:3037] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.714833 7792 raft_consensus.cc:504] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.715113 7792 leader_election.cc:288] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:53.715353 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9602786f8422d962885dda25c6c85" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:53.715368 6436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9602786f8422d962885dda25c6c85" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:53.715440 6305 raft_consensus.cc:3037] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.715451 6436 raft_consensus.cc:3037] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.716086 6305 raft_consensus.cc:2454] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:53.716162 6436 raft_consensus.cc:2454] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:53.716409 6501 leader_election.cc:302] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:53.716547 7792 raft_consensus.cc:2781] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:53.716712 7792 raft_consensus.cc:686] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:53.716826 7792 consensus_queue.cc:237] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.718415 6160 catalog_manager.cc:5484] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:53.734521 7798 raft_consensus.cc:482] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:53.734607 7798 raft_consensus.cc:504] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.734941 7798 leader_election.cc:288] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:53.735247 6566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa0b29181066483386cab4c6f921f8f0" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" is_pre_election: true
I20240503 07:59:53.735268 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa0b29181066483386cab4c6f921f8f0" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:53.735350 6566 raft_consensus.cc:2454] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:53.735399 6699 raft_consensus.cc:2454] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 0.
I20240503 07:59:53.735579 6241 leader_election.cc:302] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:53.735705 7798 raft_consensus.cc:2781] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:53.735754 7798 raft_consensus.cc:482] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:53.735780 7798 raft_consensus.cc:3037] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.736460 7798 raft_consensus.cc:504] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.736724 7798 leader_election.cc:288] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:53.736999 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa0b29181066483386cab4c6f921f8f0" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:53.737000 6566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa0b29181066483386cab4c6f921f8f0" candidate_uuid: "253dd189c3c542248402cba222394f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:53.737111 6566 raft_consensus.cc:3037] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.737118 6699 raft_consensus.cc:3037] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.737766 6699 raft_consensus.cc:2454] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:53.737772 6566 raft_consensus.cc:2454] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253dd189c3c542248402cba222394f43 in term 1.
I20240503 07:59:53.737999 6241 leader_election.cc:302] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:53.738132 7798 raft_consensus.cc:2781] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:53.738214 7798 raft_consensus.cc:686] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Becoming Leader. State: Replica: 253dd189c3c542248402cba222394f43, State: Running, Role: LEADER
I20240503 07:59:53.738327 7798 consensus_queue.cc:237] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } }
I20240503 07:59:53.739710 7792 raft_consensus.cc:482] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:53.739775 7792 raft_consensus.cc:504] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:53.739801 6160 catalog_manager.cc:5484] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 253dd189c3c542248402cba222394f43 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:53.740067 7792 leader_election.cc:288] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:53.740303 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d608390b027432f8a3f89130c051759" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43" is_pre_election: true
I20240503 07:59:53.740406 6305 raft_consensus.cc:2454] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:53.740442 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d608390b027432f8a3f89130c051759" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113" is_pre_election: true
I20240503 07:59:53.740546 6699 raft_consensus.cc:2454] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 0.
I20240503 07:59:53.740654 6502 leader_election.cc:302] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 34abd76ae3764e7f9641eaddaffdd5a5; no voters:
I20240503 07:59:53.740844 7792 raft_consensus.cc:2781] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:53.740913 7792 raft_consensus.cc:482] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:53.740968 7792 raft_consensus.cc:3037] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.741753 7792 raft_consensus.cc:504] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:53.742290 7792 leader_election.cc:288] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:53.742431 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d608390b027432f8a3f89130c051759" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:53.742431 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d608390b027432f8a3f89130c051759" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:53.742529 6305 raft_consensus.cc:3037] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.742529 6699 raft_consensus.cc:3037] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:53.743299 6699 raft_consensus.cc:2454] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:53.743299 6305 raft_consensus.cc:2454] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 1.
I20240503 07:59:53.743537 6503 leader_election.cc:302] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34abd76ae3764e7f9641eaddaffdd5a5, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 07:59:53.743690 7792 raft_consensus.cc:2781] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:53.743768 7792 raft_consensus.cc:686] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:53.743885 7792 consensus_queue.cc:237] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } }
I20240503 07:59:53.745301 6160 catalog_manager.cc:5484] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:53.857364 7915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:53.857561 7915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:53.860528 7915 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 07:59:53.943487 6699 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.943858 7792 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.944988 6698 raft_consensus.cc:1261] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.946025 6305 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.946997 7355 consensus_queue.cc:1035] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.947374 6566 raft_consensus.cc:1261] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.947772 7355 consensus_queue.cc:1035] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.948776 7911 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.948894 6436 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.949704 7945 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:53.955261 6306 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:53.961597 7792 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.243 ms
time per row: 0.0546214 ms
I20240503 07:59:54.008616 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:41285
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:42987
--webserver_port=36927
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 07:59:54.109242 7952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:54.109417 7952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:54.109467 7952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:54.112387 7952 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:54.112438 7952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:54.112515 7952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:54.116181 7952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:41285
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=36927
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.7952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:54.116953 7952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:54.118054 7952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:54.126056 7960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:54.126582 7957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:54.127579 7958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:54.128333 7952 server_base.cc:1034] running on GCE node
I20240503 07:59:54.128715 7952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:54.129251 7952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:54.130388 7952 hybrid_clock.cc:648] HybridClock initialized: now 1714723194130360 us; error 43 us; skew 500 ppm
I20240503 07:59:54.131970 7952 webserver.cc:460] Webserver started at http://127.0.99.133:36927/ using document root <none> and password file <none>
I20240503 07:59:54.132452 7952 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:54.132515 7952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:54.135555 7952 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:54.137234 7966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.138181 7952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:54.138270 7952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "61a9a0a13deb4d72b73f8623f6576766"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:54.138667 7952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:54.148447 7952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:54.148895 7952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:54.149025 7952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:54.149473 7952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:54.150202 7952 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:54.150243 7952 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.150287 7952 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:54.150312 7952 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.157835 7952 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:41285
I20240503 07:59:54.157861 8079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:41285 every 8 connection(s)
I20240503 07:59:54.158708 7952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:54.163270 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 7952
I20240503 07:59:54.163635 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:42851
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:42987
--webserver_port=40691
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:59:54.166865 8080 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:54.167052 8080 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:54.167516 8080 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:54.168512 6164 ts_manager.cc:194] Re-registered known tserver with Master: 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285)
I20240503 07:59:54.169102 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:35437
W20240503 07:59:54.266381 8083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:54.266602 8083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:54.266671 8083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:54.269734 8083 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:54.269809 8083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:54.269906 8083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:54.273380 8083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:42851
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=40691
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.8083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:54.274143 8083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:54.275104 8083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:54.282981 8088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:54.283488 8091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:54.284379 8089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:54.285184 8083 server_base.cc:1034] running on GCE node
I20240503 07:59:54.285485 8083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:54.285925 8083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:54.287073 8083 hybrid_clock.cc:648] HybridClock initialized: now 1714723194287062 us; error 22 us; skew 500 ppm
I20240503 07:59:54.288623 8083 webserver.cc:460] Webserver started at http://127.0.99.134:40691/ using document root <none> and password file <none>
I20240503 07:59:54.289114 8083 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:54.289180 8083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:54.292032 8083 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:54.293687 8097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.294615 8083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:54.294698 8083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "66f9a693c3b3409dbcce8f3ee500bf5e"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:54.295091 8083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:54.311619 8083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:54.312132 8083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:54.312275 8083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:54.312793 8083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:54.313591 8083 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:54.313633 8083 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.313679 8083 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:54.313705 8083 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.321152 8083 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42851
I20240503 07:59:54.321180 8210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42851 every 8 connection(s)
I20240503 07:59:54.322010 8083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:54.328892 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 8083
I20240503 07:59:54.329358 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:35381
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:42987
--webserver_port=36953
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:59:54.334895 8211 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:54.335083 8211 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:54.335515 8211 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:54.336407 6164 ts_manager.cc:194] Re-registered known tserver with Master: 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851)
I20240503 07:59:54.336915 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:39709
W20240503 07:59:54.438046 8214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:54.438225 8214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:54.438282 8214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:54.441190 8214 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:54.441247 8214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:54.441331 8214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:54.444854 8214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:35381
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=36953
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.8214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:54.445600 8214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:54.446626 8214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:54.454553 8220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:54.455092 8222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:54.455948 8219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:54.456579 8214 server_base.cc:1034] running on GCE node
I20240503 07:59:54.456923 8214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:54.457437 8214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:54.458576 8214 hybrid_clock.cc:648] HybridClock initialized: now 1714723194458560 us; error 27 us; skew 500 ppm
I20240503 07:59:54.460331 8214 webserver.cc:460] Webserver started at http://127.0.99.135:36953/ using document root <none> and password file <none>
I20240503 07:59:54.460853 8214 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:54.460919 8214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:54.463984 8214 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:54.465641 8228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.466526 8214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:54.466625 8214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "cae9627a7600448faa8fcb7989b307ee"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:54.467015 8214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:54.485453 8214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:54.485962 8214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:54.486106 8214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:54.486562 8214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:54.487332 8214 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:54.487376 8214 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.487416 8214 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:54.487443 8214 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.495841 8214 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:35381
I20240503 07:59:54.495880 8341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:35381 every 8 connection(s)
I20240503 07:59:54.496760 8214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:54.504119 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 8214
I20240503 07:59:54.504523 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:38867
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:42987
--webserver_port=45605
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:59:54.504549 8342 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:54.504734 8342 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:54.505283 8342 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:54.506177 6164 ts_manager.cc:194] Re-registered known tserver with Master: cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381)
I20240503 07:59:54.506681 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:42795
W20240503 07:59:54.610525 8345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:54.610739 8345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:54.610796 8345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:54.613832 8345 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:54.613883 8345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:54.613947 8345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:59:54.617507 8345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:38867
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=45605
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.8345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:54.618188 8345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:54.619119 8345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:54.627027 8351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:54.627593 8352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:54.628485 8354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:54.629222 8345 server_base.cc:1034] running on GCE node
I20240503 07:59:54.629570 8345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:54.630041 8345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:54.631176 8345 hybrid_clock.cc:648] HybridClock initialized: now 1714723194631164 us; error 27 us; skew 500 ppm
I20240503 07:59:54.632810 8345 webserver.cc:460] Webserver started at http://127.0.99.136:45605/ using document root <none> and password file <none>
I20240503 07:59:54.633314 8345 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:54.633381 8345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:54.636355 8345 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:54.638010 8360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.638859 8345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:54.638947 8345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "713b76caded84f269397165703739870"
format_stamp: "Formatted at 2024-05-03 07:59:44 on dist-test-slave-7b05"
I20240503 07:59:54.639286 8345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:54.655464 8345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:54.655938 8345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:54.656075 8345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:54.656487 8345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:54.657282 8345 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:54.657325 8345 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.657371 8345 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:54.657394 8345 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:54.664868 8345 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:38867
I20240503 07:59:54.664994 8473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:38867 every 8 connection(s)
I20240503 07:59:54.666818 8345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:59:54.669869 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 8345
I20240503 07:59:54.675338 8474 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:54.675515 8474 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:54.676023 8474 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:54.676896 6164 ts_manager.cc:194] Re-registered known tserver with Master: 713b76caded84f269397165703739870 (127.0.99.136:38867)
I20240503 07:59:54.677630 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:60687
W20240503 07:59:54.785020 8478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:54.785272 8478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:54.788147 8478 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:54.903573 6415 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:54.908946 8145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.909952 8014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.917145 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.930948 8407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.932291 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:54.932448 6546 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:54.953378 6546 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:54.957183 6284 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:54.959583 6415 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:54.966197 8145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.973605 8407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:54.990059 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.005674 6284 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.008423 8014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.012146 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:55.111407 6415 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.114537 6546 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.115029 6284 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.122622 8407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.129025 8014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.129632 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:55.134851 8145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.147838 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.170724 8080 heartbeater.cc:503] Master 127.0.99.190:42987 was elected leader, sending a full tablet report...
W20240503 07:59:55.223065 8591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:55.223251 8591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:55.226152 8591 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:55.276152 6284 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.282246 8014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.282476 6415 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.296633 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.298326 8407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.301595 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:55.302151 6546 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.304383 8145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.338563 8211 heartbeater.cc:503] Master 127.0.99.190:42987 was elected leader, sending a full tablet report...
I20240503 07:59:55.370290 6284 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.376334 6415 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.379402 8407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.391795 8145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.400810 8014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.404150 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:55.409464 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.415845 6546 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.439605 6699 consensus_queue.cc:237] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:55.441653 6566 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.441779 6305 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.442123 8623 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.442503 8623 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.445408 8623 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.447909 6305 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
W20240503 07:59:55.449293 6635 consensus_peers.cc:485] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7012d8274214cf58c70d52062474b3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.454180 6156 catalog_manager.cc:5484] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.454618 6566 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.460657 6566 consensus_queue.cc:237] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:55.462297 6436 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.462716 6699 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.462800 8730 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.463079 7945 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.465641 7945 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
W20240503 07:59:55.466174 6503 consensus_peers.cc:485] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd0045edfdb441f2ae6920b8a97555bb. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.466542 6436 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.468837 6164 catalog_manager.cc:5484] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.469096 6699 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.476814 6699 consensus_queue.cc:237] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:55.478077 6305 raft_consensus.cc:1261] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.478300 6436 raft_consensus.cc:1261] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.478421 8623 consensus_queue.cc:1035] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.478624 8717 consensus_queue.cc:1035] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.480165 8623 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.480563 6436 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.482162 6305 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.482867 6155 catalog_manager.cc:5484] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:55.483439 6635 consensus_peers.cc:485] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fb6518dced248a88edde5cbbb25350d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.490715 6305 consensus_queue.cc:237] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:55.492028 6436 raft_consensus.cc:1261] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.493237 8574 consensus_queue.cc:1035] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.493849 6699 raft_consensus.cc:1261] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.494400 8574 consensus_queue.cc:1035] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.498306 7355 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.500267 6437 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.500820 6164 catalog_manager.cc:5484] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:55.501515 6240 consensus_peers.cc:485] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 957ae2f17c4c41d28287d5490c54572e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.507077 6698 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.508656 8342 heartbeater.cc:503] Master 127.0.99.190:42987 was elected leader, sending a full tablet report...
I20240503 07:59:55.518261 6566 consensus_queue.cc:237] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:55.519798 6305 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.520004 6437 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.520192 8730 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.520700 8730 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.523372 8730 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
W20240503 07:59:55.523815 6504 consensus_peers.cc:485] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f40fae73f8f4c63b2d656ac880799e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.525328 6305 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.525849 6437 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.526801 6160 catalog_manager.cc:5484] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.537854 6305 consensus_queue.cc:237] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:55.539199 6437 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.539544 8574 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.540731 6566 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.541155 8574 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.541754 8574 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.543498 6566 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.544363 6437 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
W20240503 07:59:55.545658 6242 consensus_peers.cc:485] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e03d7158b64f434a8871f009864a0b72. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.546367 6155 catalog_manager.cc:5484] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.556926 6566 consensus_queue.cc:237] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:55.558334 6437 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:55.558411 6503 consensus_peers.cc:485] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9023b7fb87a4adaa91995b883df8826. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.558610 6305 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.558712 7945 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.559000 8729 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.560231 8761 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.560674 6437 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.561270 6305 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.562788 6160 catalog_manager.cc:5484] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.571453 6437 consensus_queue.cc:237] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:55.572851 6698 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.573031 6305 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.573300 8771 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.573462 7884 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.575068 7884 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.575534 6305 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.577723 6164 catalog_manager.cc:5484] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:55.581391 6371 consensus_peers.cc:485] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4925e30c2fb74934bbd5eb7415193c96. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.583537 6698 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.586961 6699 consensus_queue.cc:237] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:55.588261 6437 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.588258 6566 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.588627 8717 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.588951 8623 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:55.589123 6635 consensus_peers.cc:485] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8efd24415e484e2a9af88b3aef6f78cc. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.590545 8717 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.591002 6566 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.593187 6437 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.593683 6155 catalog_manager.cc:5484] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.600224 6305 consensus_queue.cc:237] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:55.601382 6699 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.601620 6437 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:55.601794 6240 consensus_peers.cc:485] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 387f6b1606b94e11897372c0557870e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.602015 8749 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.602198 7355 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.603783 8574 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.604142 6699 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.606066 6164 catalog_manager.cc:5484] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.607028 6437 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.614009 6699 consensus_queue.cc:237] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:55.615235 6305 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:55.615527 6635 consensus_peers.cc:485] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64eb15b0505a4649a0043be00250c299. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.615685 8623 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.615757 6566 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.616120 8623 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.617288 8718 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.617673 6305 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.619566 6155 catalog_manager.cc:5484] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.619612 6566 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
W20240503 07:59:55.623214 8742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:55.624275 8742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:55.627864 8742 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:55.629187 6437 consensus_queue.cc:237] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:55.630586 6305 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.630674 6699 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.630911 7884 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.631064 8771 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.632759 7884 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.633225 6699 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
W20240503 07:59:55.634424 6372 consensus_peers.cc:485] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9e91e1b70504ac695eae61f72d3c149. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.635620 6155 catalog_manager.cc:5484] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.637001 6305 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.645975 6437 consensus_queue.cc:237] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:55.647382 6699 raft_consensus.cc:1261] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:55.647526 6371 consensus_peers.cc:485] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82f6c17706cc4a40827501ecb7c3ebce. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.647567 6566 raft_consensus.cc:1261] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.647748 8771 consensus_queue.cc:1035] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.647902 8762 consensus_queue.cc:1035] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.650369 8771 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.650743 6699 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.650821 6566 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.653048 6155 catalog_manager.cc:5484] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.660486 6437 consensus_queue.cc:237] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:55.661707 6699 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.661864 6566 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.662170 8771 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.664317 8771 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:55.665971 6373 consensus_peers.cc:485] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37c8748a1c9d42b593b2381a1a870d2e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.673358 7884 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.673842 6699 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.676067 6566 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.676518 6164 catalog_manager.cc:5484] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.679566 8474 heartbeater.cc:503] Master 127.0.99.190:42987 was elected leader, sending a full tablet report...
I20240503 07:59:55.688077 6437 consensus_queue.cc:237] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:55.689298 6566 raft_consensus.cc:1261] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.689313 6305 raft_consensus.cc:1261] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:55.689623 6372 consensus_peers.cc:485] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a1dab998bb84cc9880252383cb1e982. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.689674 7884 consensus_queue.cc:1035] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.689814 8771 consensus_queue.cc:1035] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.691669 8771 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.692217 6305 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.692374 6566 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.695013 6155 catalog_manager.cc:5484] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.705708 6437 consensus_queue.cc:237] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:55.708312 6566 raft_consensus.cc:1261] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.708513 6699 raft_consensus.cc:1261] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.708725 8762 consensus_queue.cc:1035] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.709877 8762 consensus_queue.cc:1035] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:55.711575 6373 consensus_peers.cc:485] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a934439594245d79ed5980ff9c03277. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.712524 8762 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.713850 6699 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.716568 6164 catalog_manager.cc:5484] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.717265 6566 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:55.730305 6437 consensus_queue.cc:237] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
W20240503 07:59:55.731724 6373 consensus_peers.cc:485] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d74a03a296dc48e582faeddc7b978ccd. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.731740 6699 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.731772 6305 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.732048 8771 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.734318 8771 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.734658 8771 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.735162 6699 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.736678 6305 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.738399 6164 catalog_manager.cc:5484] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:55.746186 6284 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.746484 6415 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.752055 8014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.757983 6546 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:55.758975 6566 consensus_queue.cc:237] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:55.759126 8145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.761024 6305 raft_consensus.cc:1261] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.761426 8730 consensus_queue.cc:1035] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.762032 6436 raft_consensus.cc:1261] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.762374 8730 consensus_queue.cc:1035] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:55.764401 6502 consensus_peers.cc:485] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d8a5e7a21684a4782d43c5a1131f652. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.768378 8730 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.771219 6305 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.771647 6160 catalog_manager.cc:5484] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.773344 6436 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:55.776547 8407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.778153 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:55.796240 6566 consensus_queue.cc:237] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:55.797828 6436 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:55.797873 6503 consensus_peers.cc:485] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8625e8b653624ea6acf6e04c3fd352a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:55.798043 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:55.798301 8761 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.801940 6699 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:55.802318 8761 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:55.803143 7945 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.803598 6436 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:55.805671 6163 catalog_manager.cc:5484] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:55.809813 6699 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
Runtime error: ksck discovered errors
I20240503 07:59:55.893301 8996 ts_tablet_manager.cc:902] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:55.894475 8996 tablet_copy_client.cc:308] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 07:59:55.901343 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet d7012d8274214cf58c70d52062474b3a from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:54799)
I20240503 07:59:55.901472 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet d7012d8274214cf58c70d52062474b3a from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:54799: session id = 61a9a0a13deb4d72b73f8623f6576766-d7012d8274214cf58c70d52062474b3a
I20240503 07:59:55.902714 6719 tablet_copy_source_session.cc:205] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.905762 8996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7012d8274214cf58c70d52062474b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.911113 8996 tablet_copy_client.cc:783] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.911291 8996 tablet_copy_client.cc:647] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.912544 9003 tablet_copy_client.cc:1002] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:55.921883 9024 ts_tablet_manager.cc:902] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:55.922515 9024 tablet_copy_client.cc:308] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 07:59:55.923759 9030 ts_tablet_manager.cc:902] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:55.924177 9030 tablet_copy_client.cc:308] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 07:59:55.926486 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet 2fb6518dced248a88edde5cbbb25350d from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:47971)
I20240503 07:59:55.926584 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet 2fb6518dced248a88edde5cbbb25350d from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:47971: session id = 713b76caded84f269397165703739870-2fb6518dced248a88edde5cbbb25350d
I20240503 07:59:55.927436 6719 tablet_copy_source_session.cc:205] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.928763 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet e03d7158b64f434a8871f009864a0b72 from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:52377)
I20240503 07:59:55.928905 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet e03d7158b64f434a8871f009864a0b72 from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:52377: session id = 713b76caded84f269397165703739870-e03d7158b64f434a8871f009864a0b72
I20240503 07:59:55.929939 9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb6518dced248a88edde5cbbb25350d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.930225 6326 tablet_copy_source_session.cc:205] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.931185 9030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03d7158b64f434a8871f009864a0b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.935528 9030 tablet_copy_client.cc:783] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.935743 9024 tablet_copy_client.cc:783] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.935740 9030 tablet_copy_client.cc:647] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.935894 9024 tablet_copy_client.cc:647] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.936937 9028 tablet_copy_client.cc:1002] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:55.937353 9033 tablet_copy_client.cc:1002] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
W20240503 07:59:55.960804 8956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:55.960989 8956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:55.964088 8956 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:55.980067 9046 ts_tablet_manager.cc:902] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:55.980513 9046 tablet_copy_client.cc:308] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 07:59:55.980934 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet 8efd24415e484e2a9af88b3aef6f78cc from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:54799)
I20240503 07:59:55.981025 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet 8efd24415e484e2a9af88b3aef6f78cc from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:54799: session id = 61a9a0a13deb4d72b73f8623f6576766-8efd24415e484e2a9af88b3aef6f78cc
I20240503 07:59:55.981956 6719 tablet_copy_source_session.cc:205] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:55.982652 9046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efd24415e484e2a9af88b3aef6f78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:55.984570 9046 tablet_copy_client.cc:783] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:55.984757 9046 tablet_copy_client.cc:647] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:55.985850 9049 tablet_copy_client.cc:1002] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.019237 9071 ts_tablet_manager.cc:902] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:56.019975 9071 tablet_copy_client.cc:308] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 07:59:56.024806 6457 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet 4925e30c2fb74934bbd5eb7415193c96 from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:36419)
I20240503 07:59:56.024933 6457 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet 4925e30c2fb74934bbd5eb7415193c96 from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:36419: session id = cae9627a7600448faa8fcb7989b307ee-4925e30c2fb74934bbd5eb7415193c96
I20240503 07:59:56.026297 6457 tablet_copy_source_session.cc:205] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.029327 9071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4925e30c2fb74934bbd5eb7415193c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.034381 9071 tablet_copy_client.cc:783] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.034546 9071 tablet_copy_client.cc:647] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.036083 9074 tablet_copy_client.cc:1002] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.054302 6546 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:56.054391 6284 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:56.055773 9088 ts_tablet_manager.cc:902] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:56.056496 9088 tablet_copy_client.cc:308] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 07:59:56.057186 6415 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:56.062003 8145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.064550 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.065781 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet cd0045edfdb441f2ae6920b8a97555bb from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:35513)
I20240503 07:59:56.065905 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet cd0045edfdb441f2ae6920b8a97555bb from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:35513: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-cd0045edfdb441f2ae6920b8a97555bb
I20240503 07:59:56.067184 6588 tablet_copy_source_session.cc:205] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.067862 9094 ts_tablet_manager.cc:902] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:56.068966 9094 tablet_copy_client.cc:308] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 07:59:56.069196 8407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.070616 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:56.072160 9088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd0045edfdb441f2ae6920b8a97555bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.076128 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet 957ae2f17c4c41d28287d5490c54572e from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:57621)
I20240503 07:59:56.076224 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet 957ae2f17c4c41d28287d5490c54572e from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:57621: session id = cae9627a7600448faa8fcb7989b307ee-957ae2f17c4c41d28287d5490c54572e
I20240503 07:59:56.077138 6326 tablet_copy_source_session.cc:205] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.077891 9094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957ae2f17c4c41d28287d5490c54572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.079802 9094 tablet_copy_client.cc:783] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.079924 9094 tablet_copy_client.cc:647] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.081737 9088 tablet_copy_client.cc:783] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.081940 9088 tablet_copy_client.cc:647] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.083387 9092 tablet_copy_client.cc:1002] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.097620 9096 tablet_copy_client.cc:1002] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.102284 9100 ts_tablet_manager.cc:902] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:56.102855 9100 tablet_copy_client.cc:308] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 07:59:56.104532 8014 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:56.107474 6457 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet 9a1dab998bb84cc9880252383cb1e982 from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:54879)
I20240503 07:59:56.107565 6457 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet 9a1dab998bb84cc9880252383cb1e982 from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:54879: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-9a1dab998bb84cc9880252383cb1e982
I20240503 07:59:56.108404 6457 tablet_copy_source_session.cc:205] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.109371 9106 ts_tablet_manager.cc:902] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:56.109891 9106 tablet_copy_client.cc:308] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 07:59:56.110280 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet 64eb15b0505a4649a0043be00250c299 from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:47971)
I20240503 07:59:56.110378 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet 64eb15b0505a4649a0043be00250c299 from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:47971: session id = 713b76caded84f269397165703739870-64eb15b0505a4649a0043be00250c299
I20240503 07:59:56.111253 6719 tablet_copy_source_session.cc:205] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.112102 9106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64eb15b0505a4649a0043be00250c299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.114323 9106 tablet_copy_client.cc:783] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.114497 9106 tablet_copy_client.cc:647] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.115375 9110 tablet_copy_client.cc:1002] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.115828 9100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1dab998bb84cc9880252383cb1e982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.116659 9099 ts_tablet_manager.cc:902] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:56.116864 9105 ts_tablet_manager.cc:902] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:56.117847 9100 tablet_copy_client.cc:783] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.117870 9099 tablet_copy_client.cc:308] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 07:59:56.117970 9100 tablet_copy_client.cc:647] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.118510 6457 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet d9e91e1b70504ac695eae61f72d3c149 from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:54879)
I20240503 07:59:56.118600 6457 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet d9e91e1b70504ac695eae61f72d3c149 from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:54879: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-d9e91e1b70504ac695eae61f72d3c149
I20240503 07:59:56.119364 6457 tablet_copy_source_session.cc:205] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.120112 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e91e1b70504ac695eae61f72d3c149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.121697 9101 tablet_copy_client.cc:1002] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.121896 9099 tablet_copy_client.cc:783] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.122071 9099 tablet_copy_client.cc:647] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.122769 9124 ts_tablet_manager.cc:902] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:56.123183 9124 tablet_copy_client.cc:308] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 07:59:56.123577 9105 tablet_copy_client.cc:308] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 07:59:56.123594 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet a9023b7fb87a4adaa91995b883df8826 from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:35513)
I20240503 07:59:56.123678 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet a9023b7fb87a4adaa91995b883df8826 from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:35513: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-a9023b7fb87a4adaa91995b883df8826
I20240503 07:59:56.124410 6588 tablet_copy_source_session.cc:205] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.125051 9124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9023b7fb87a4adaa91995b883df8826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.126624 9124 tablet_copy_client.cc:783] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.126849 9124 tablet_copy_client.cc:647] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.127666 9128 tablet_copy_client.cc:1002] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.128355 9115 tablet_copy_client.cc:1002] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.128851 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 2f40fae73f8f4c63b2d656ac880799e2 from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:45731)
I20240503 07:59:56.128930 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 2f40fae73f8f4c63b2d656ac880799e2 from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:45731: session id = 61a9a0a13deb4d72b73f8623f6576766-2f40fae73f8f4c63b2d656ac880799e2
I20240503 07:59:56.129643 6588 tablet_copy_source_session.cc:205] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.129887 9129 ts_tablet_manager.cc:902] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:56.130892 9129 tablet_copy_client.cc:308] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 07:59:56.131299 6456 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet 82f6c17706cc4a40827501ecb7c3ebce from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:36419)
I20240503 07:59:56.131378 6456 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet 82f6c17706cc4a40827501ecb7c3ebce from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:36419: session id = cae9627a7600448faa8fcb7989b307ee-82f6c17706cc4a40827501ecb7c3ebce
I20240503 07:59:56.132184 6456 tablet_copy_source_session.cc:205] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.133368 9129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f6c17706cc4a40827501ecb7c3ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.133539 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f40fae73f8f4c63b2d656ac880799e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.135506 9129 tablet_copy_client.cc:783] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.135637 9129 tablet_copy_client.cc:647] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.135649 9105 tablet_copy_client.cc:783] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.135790 9105 tablet_copy_client.cc:647] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.136706 9114 tablet_copy_client.cc:1002] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.137435 9139 ts_tablet_manager.cc:902] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:56.137964 9131 tablet_copy_client.cc:1002] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.137998 9139 tablet_copy_client.cc:308] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 07:59:56.138362 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet 387f6b1606b94e11897372c0557870e4 from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:57621)
I20240503 07:59:56.138461 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet 387f6b1606b94e11897372c0557870e4 from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:57621: session id = cae9627a7600448faa8fcb7989b307ee-387f6b1606b94e11897372c0557870e4
I20240503 07:59:56.139313 6326 tablet_copy_source_session.cc:205] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.140096 9139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387f6b1606b94e11897372c0557870e4. 1 dirs total, 0 dirs full, 0 dirs failed
Runtime error: ksck discovered errors
I20240503 07:59:56.143558 9139 tablet_copy_client.cc:783] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.143671 9139 tablet_copy_client.cc:647] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.144790 9143 tablet_copy_client.cc:1002] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.151399 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.137:0
--local_ip_for_outbound_sockets=127.0.99.137
--webserver_interface=127.0.99.137
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42987
--builtin_ntp_servers=127.0.99.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 07:59:56.225600 9215 ts_tablet_manager.cc:902] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:56.226742 9215 tablet_copy_client.cc:308] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 07:59:56.230027 6456 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet d74a03a296dc48e582faeddc7b978ccd from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:39381)
I20240503 07:59:56.230113 6456 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet d74a03a296dc48e582faeddc7b978ccd from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:39381: session id = 61a9a0a13deb4d72b73f8623f6576766-d74a03a296dc48e582faeddc7b978ccd
I20240503 07:59:56.230748 6456 tablet_copy_source_session.cc:205] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.231429 9215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74a03a296dc48e582faeddc7b978ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.233845 9215 tablet_copy_client.cc:783] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.233994 9215 tablet_copy_client.cc:647] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.235450 9217 tablet_copy_client.cc:1002] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.246593 9235 ts_tablet_manager.cc:902] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:56.247150 9235 tablet_copy_client.cc:308] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 07:59:56.247215 9240 ts_tablet_manager.cc:902] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:56.247689 9240 tablet_copy_client.cc:308] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 07:59:56.248122 6456 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet 37c8748a1c9d42b593b2381a1a870d2e from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:39381)
I20240503 07:59:56.248210 6456 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet 37c8748a1c9d42b593b2381a1a870d2e from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:39381: session id = 61a9a0a13deb4d72b73f8623f6576766-37c8748a1c9d42b593b2381a1a870d2e
I20240503 07:59:56.248950 6456 tablet_copy_source_session.cc:205] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.249657 9240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c8748a1c9d42b593b2381a1a870d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.251250 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 3d8a5e7a21684a4782d43c5a1131f652 from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:56655)
I20240503 07:59:56.251350 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 3d8a5e7a21684a4782d43c5a1131f652 from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:56655: session id = cae9627a7600448faa8fcb7989b307ee-3d8a5e7a21684a4782d43c5a1131f652
I20240503 07:59:56.251482 9240 tablet_copy_client.cc:783] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.251638 9240 tablet_copy_client.cc:647] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.252179 6588 tablet_copy_source_session.cc:205] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.252590 9242 tablet_copy_client.cc:1002] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.253209 9245 ts_tablet_manager.cc:902] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:56.253214 9235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8a5e7a21684a4782d43c5a1131f652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.253669 9245 tablet_copy_client.cc:308] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 07:59:56.254052 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 8625e8b653624ea6acf6e04c3fd352a4 from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:35513)
I20240503 07:59:56.254130 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 8625e8b653624ea6acf6e04c3fd352a4 from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:35513: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-8625e8b653624ea6acf6e04c3fd352a4
I20240503 07:59:56.254703 6588 tablet_copy_source_session.cc:205] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.255370 9245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8625e8b653624ea6acf6e04c3fd352a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.255700 9235 tablet_copy_client.cc:783] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.255831 9235 tablet_copy_client.cc:647] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.256775 9239 tablet_copy_client.cc:1002] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.257226 9245 tablet_copy_client.cc:783] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.257360 9245 tablet_copy_client.cc:647] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.258251 9249 tablet_copy_client.cc:1002] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:56.260540 9250 ts_tablet_manager.cc:902] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:56.260954 9250 tablet_copy_client.cc:308] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 07:59:56.264344 6456 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet 8a934439594245d79ed5980ff9c03277 from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:40651)
I20240503 07:59:56.264420 6456 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet 8a934439594245d79ed5980ff9c03277 from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:40651: session id = 713b76caded84f269397165703739870-8a934439594245d79ed5980ff9c03277
I20240503 07:59:56.264962 6456 tablet_copy_source_session.cc:205] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:56.265690 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a934439594245d79ed5980ff9c03277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:56.267138 9250 tablet_copy_client.cc:783] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:56.267289 9250 tablet_copy_client.cc:647] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:56.268184 9254 tablet_copy_client.cc:1002] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
W20240503 07:59:56.274086 9154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:56.274247 9154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:56.274314 9154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:56.277216 9154 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:56.277271 9154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:56.277359 9154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.137
I20240503 07:59:56.280933 9154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.137:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.0.99.137
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:42987
--never_fsync=true
--heap_profile_path=/tmp/kudu.9154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.137
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:56.281756 9154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:56.282893 9154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:56.290702 9259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:56.291214 9262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:56.292122 9260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:56.292809 9154 server_base.cc:1034] running on GCE node
I20240503 07:59:56.293159 9154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:56.293646 9154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:56.294787 9154 hybrid_clock.cc:648] HybridClock initialized: now 1714723196294769 us; error 29 us; skew 500 ppm
I20240503 07:59:56.296541 9154 webserver.cc:460] Webserver started at http://127.0.99.137:35581/ using document root <none> and password file <none>
I20240503 07:59:56.296996 9154 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:56.297092 9154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:56.297274 9154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:56.298713 9154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "0394d429c78744aca1220af8f39b11c1"
format_stamp: "Formatted at 2024-05-03 07:59:56 on dist-test-slave-7b05"
I20240503 07:59:56.299100 9154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "0394d429c78744aca1220af8f39b11c1"
format_stamp: "Formatted at 2024-05-03 07:59:56 on dist-test-slave-7b05"
I20240503 07:59:56.301868 9154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:56.303707 9268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:56.304574 9154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:56.304656 9154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "0394d429c78744aca1220af8f39b11c1"
format_stamp: "Formatted at 2024-05-03 07:59:56 on dist-test-slave-7b05"
I20240503 07:59:56.304734 9154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:56.315593 9154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:56.316025 9154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:56.316148 9154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:56.316584 9154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:56.317329 9154 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:56.317378 9154 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:56.317425 9154 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:56.317452 9154 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:56.325325 9154 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.137:39867
I20240503 07:59:56.325359 9381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.137:39867 every 8 connection(s)
I20240503 07:59:56.326201 9154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 07:59:56.329126 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 9154
I20240503 07:59:56.329275 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 07:59:56.333712 9382 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42987
I20240503 07:59:56.333890 9382 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:56.334357 9382 heartbeater.cc:511] Master 127.0.99.190:42987 requested a full tablet report, sending...
I20240503 07:59:56.335148 6164 ts_manager.cc:194] Registered new tserver with Master: 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867)
I20240503 07:59:56.335634 6164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.137:60459
I20240503 07:59:57.337179 9382 heartbeater.cc:503] Master 127.0.99.190:42987 was elected leader, sending a full tablet report...
I20240503 07:59:57.412933 8996 tablet_copy_client.cc:523] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.414749 8996 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 07:59:57.419692 8996 log.cc:824] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:57.437362 9024 tablet_copy_client.cc:523] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.437570 9030 tablet_copy_client.cc:523] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.439203 9024 tablet_bootstrap.cc:492] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 07:59:57.439203 9030 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 07:59:57.440099 8996 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.440395 8996 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 07:59:57.440701 8996 ts_tablet_manager.cc:1367] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.026s user 0.027s sys 0.000s
I20240503 07:59:57.442986 8996 raft_consensus.cc:348] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.443204 8996 raft_consensus.cc:729] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 07:59:57.443542 8996 consensus_queue.cc:260] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.444342 8996 ts_tablet_manager.cc:1398] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:57.444921 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-d7012d8274214cf58c70d52062474b3a received from {username='slave'} at 127.0.99.133:54799
I20240503 07:59:57.445020 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-d7012d8274214cf58c70d52062474b3a on tablet d7012d8274214cf58c70d52062474b3a with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 07:59:57.445451 9024 log.cc:824] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:57.463544 9030 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.463583 9024 tablet_bootstrap.cc:492] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.463801 9024 tablet_bootstrap.cc:492] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 07:59:57.463811 9030 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 07:59:57.464043 9024 ts_tablet_manager.cc:1367] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.000s
I20240503 07:59:57.464043 9030 ts_tablet_manager.cc:1367] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.025s user 0.025s sys 0.000s
I20240503 07:59:57.465778 9030 raft_consensus.cc:348] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:57.465778 9024 raft_consensus.cc:348] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:57.465986 9024 raft_consensus.cc:729] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 07:59:57.465986 9030 raft_consensus.cc:729] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 07:59:57.466270 9030 consensus_queue.cc:260] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:57.466270 9024 consensus_queue.cc:260] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:57.466943 9024 ts_tablet_manager.cc:1398] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:57.467490 9030 ts_tablet_manager.cc:1398] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:57.467561 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session 713b76caded84f269397165703739870-2fb6518dced248a88edde5cbbb25350d received from {username='slave'} at 127.0.99.136:47971
I20240503 07:59:57.467624 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session 713b76caded84f269397165703739870-2fb6518dced248a88edde5cbbb25350d on tablet 2fb6518dced248a88edde5cbbb25350d with peer 713b76caded84f269397165703739870
I20240503 07:59:57.468430 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 713b76caded84f269397165703739870-e03d7158b64f434a8871f009864a0b72 received from {username='slave'} at 127.0.99.136:52377
I20240503 07:59:57.468523 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 713b76caded84f269397165703739870-e03d7158b64f434a8871f009864a0b72 on tablet e03d7158b64f434a8871f009864a0b72 with peer 713b76caded84f269397165703739870
I20240503 07:59:57.486115 9046 tablet_copy_client.cc:523] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.487210 9046 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 07:59:57.504485 9046 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.504674 9046 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 07:59:57.504787 9046 ts_tablet_manager.cc:1367] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240503 07:59:57.505242 9046 raft_consensus.cc:348] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.505336 9046 raft_consensus.cc:729] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 07:59:57.505403 9046 consensus_queue.cc:260] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.505610 9046 ts_tablet_manager.cc:1398] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:57.506029 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-8efd24415e484e2a9af88b3aef6f78cc received from {username='slave'} at 127.0.99.133:54799
I20240503 07:59:57.506124 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-8efd24415e484e2a9af88b3aef6f78cc on tablet 8efd24415e484e2a9af88b3aef6f78cc with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 07:59:57.536473 9071 tablet_copy_client.cc:523] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.538323 9071 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 07:59:57.544054 9071 log.cc:824] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:57.555617 8428 raft_consensus.cc:1203] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.558250 8034 raft_consensus.cc:1203] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.563820 9071 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.564093 9071 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 07:59:57.564352 9071 ts_tablet_manager.cc:1367] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.001s
I20240503 07:59:57.566160 9071 raft_consensus.cc:348] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.566367 9071 raft_consensus.cc:729] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 07:59:57.566663 9071 consensus_queue.cc:260] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.567309 9071 ts_tablet_manager.cc:1398] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240503 07:59:57.568650 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-4925e30c2fb74934bbd5eb7415193c96 received from {username='slave'} at 127.0.99.135:36419
I20240503 07:59:57.568734 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-4925e30c2fb74934bbd5eb7415193c96 on tablet 4925e30c2fb74934bbd5eb7415193c96 with peer cae9627a7600448faa8fcb7989b307ee
I20240503 07:59:57.583727 9088 tablet_copy_client.cc:523] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.585422 9088 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 07:59:57.590137 9088 log.cc:824] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:57.593690 8296 raft_consensus.cc:1203] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.598310 9094 tablet_copy_client.cc:523] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.599534 9094 tablet_bootstrap.cc:492] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 07:59:57.610735 9088 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.611040 9088 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 07:59:57.611308 9088 ts_tablet_manager.cc:1367] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240503 07:59:57.613622 9088 raft_consensus.cc:348] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.613864 9088 raft_consensus.cc:729] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 07:59:57.614253 9088 consensus_queue.cc:260] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.615175 9088 ts_tablet_manager.cc:1398] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 07:59:57.615773 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-cd0045edfdb441f2ae6920b8a97555bb received from {username='slave'} at 127.0.99.134:35513
I20240503 07:59:57.615866 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-cd0045edfdb441f2ae6920b8a97555bb on tablet cd0045edfdb441f2ae6920b8a97555bb with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 07:59:57.618227 9106 tablet_copy_client.cc:523] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.618809 9094 tablet_bootstrap.cc:492] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.618993 9094 tablet_bootstrap.cc:492] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 07:59:57.619122 9094 ts_tablet_manager.cc:1367] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240503 07:59:57.619520 9094 raft_consensus.cc:348] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.619601 9094 raft_consensus.cc:729] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 07:59:57.619607 9106 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 07:59:57.619666 9094 consensus_queue.cc:260] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.619869 9094 ts_tablet_manager.cc:1398] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:57.620381 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-957ae2f17c4c41d28287d5490c54572e received from {username='slave'} at 127.0.99.135:57621
I20240503 07:59:57.620448 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-957ae2f17c4c41d28287d5490c54572e on tablet 957ae2f17c4c41d28287d5490c54572e with peer cae9627a7600448faa8fcb7989b307ee
I20240503 07:59:57.621955 9100 tablet_copy_client.cc:523] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.623121 9100 tablet_bootstrap.cc:492] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 07:59:57.627948 9124 tablet_copy_client.cc:523] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.628609 9099 tablet_copy_client.cc:523] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.629174 9124 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 07:59:57.629787 9099 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 07:59:57.632735 8034 raft_consensus.cc:1203] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.636958 9105 tablet_copy_client.cc:523] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.638213 9129 tablet_copy_client.cc:523] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.638231 9105 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 07:59:57.639470 9129 tablet_bootstrap.cc:492] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 07:59:57.645648 9139 tablet_copy_client.cc:523] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.646795 9139 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 07:59:57.649231 8296 raft_consensus.cc:1203] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.658569 8165 raft_consensus.cc:1203] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.660494 9099 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.660800 9099 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 07:59:57.660996 9099 ts_tablet_manager.cc:1367] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.000s
I20240503 07:59:57.661568 9099 raft_consensus.cc:348] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.661754 9099 raft_consensus.cc:729] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 07:59:57.661895 9099 consensus_queue.cc:260] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.663540 9099 ts_tablet_manager.cc:1398] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:57.669804 9100 tablet_bootstrap.cc:492] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.670049 9100 tablet_bootstrap.cc:492] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 07:59:57.670199 9100 ts_tablet_manager.cc:1367] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.047s user 0.019s sys 0.006s
I20240503 07:59:57.670568 9100 raft_consensus.cc:348] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.670677 9100 raft_consensus.cc:729] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 07:59:57.670768 9100 consensus_queue.cc:260] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.674104 9129 tablet_bootstrap.cc:492] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.674449 9129 tablet_bootstrap.cc:492] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 07:59:57.674635 9129 ts_tablet_manager.cc:1367] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.000s
I20240503 07:59:57.675035 9129 raft_consensus.cc:348] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.675194 9129 raft_consensus.cc:729] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 07:59:57.675325 9129 consensus_queue.cc:260] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.675637 9129 ts_tablet_manager.cc:1398] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:57.676245 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-82f6c17706cc4a40827501ecb7c3ebce received from {username='slave'} at 127.0.99.135:36419
I20240503 07:59:57.676339 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-82f6c17706cc4a40827501ecb7c3ebce on tablet 82f6c17706cc4a40827501ecb7c3ebce with peer cae9627a7600448faa8fcb7989b307ee
I20240503 07:59:57.680924 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-d9e91e1b70504ac695eae61f72d3c149 received from {username='slave'} at 127.0.99.134:54879
I20240503 07:59:57.681001 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-d9e91e1b70504ac695eae61f72d3c149 on tablet d9e91e1b70504ac695eae61f72d3c149 with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 07:59:57.691797 9106 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.692054 9106 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 07:59:57.692397 9100 ts_tablet_manager.cc:1398] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240503 07:59:57.692768 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-9a1dab998bb84cc9880252383cb1e982 received from {username='slave'} at 127.0.99.134:54879
I20240503 07:59:57.692842 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-9a1dab998bb84cc9880252383cb1e982 on tablet 9a1dab998bb84cc9880252383cb1e982 with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 07:59:57.695246 9124 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.695551 9124 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 07:59:57.695714 9124 ts_tablet_manager.cc:1367] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.067s user 0.022s sys 0.004s
I20240503 07:59:57.696097 9124 raft_consensus.cc:348] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.696210 9124 raft_consensus.cc:729] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 07:59:57.696300 9124 consensus_queue.cc:260] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.697355 9106 ts_tablet_manager.cc:1367] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.078s user 0.026s sys 0.000s
I20240503 07:59:57.697733 9139 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.698060 9139 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 07:59:57.698256 9139 ts_tablet_manager.cc:1367] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.051s user 0.020s sys 0.004s
I20240503 07:59:57.698637 9124 ts_tablet_manager.cc:1398] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:57.698633 9139 raft_consensus.cc:348] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.698777 9139 raft_consensus.cc:729] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 07:59:57.698884 9139 consensus_queue.cc:260] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.699180 9139 ts_tablet_manager.cc:1398] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:57.699507 9105 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.699755 9105 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 07:59:57.699725 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-a9023b7fb87a4adaa91995b883df8826 received from {username='slave'} at 127.0.99.134:35513
I20240503 07:59:57.699796 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-a9023b7fb87a4adaa91995b883df8826 on tablet a9023b7fb87a4adaa91995b883df8826 with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 07:59:57.700158 9105 ts_tablet_manager.cc:1367] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.062s user 0.022s sys 0.004s
I20240503 07:59:57.700709 9105 raft_consensus.cc:348] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.700817 9105 raft_consensus.cc:729] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 07:59:57.700918 9105 consensus_queue.cc:260] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.701174 9105 ts_tablet_manager.cc:1398] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:57.701471 9106 raft_consensus.cc:348] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:57.701576 9106 raft_consensus.cc:729] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 07:59:57.701678 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-2f40fae73f8f4c63b2d656ac880799e2 received from {username='slave'} at 127.0.99.133:45731
I20240503 07:59:57.701668 9106 consensus_queue.cc:260] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:57.701737 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-2f40fae73f8f4c63b2d656ac880799e2 on tablet 2f40fae73f8f4c63b2d656ac880799e2 with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 07:59:57.701913 9106 ts_tablet_manager.cc:1398] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240503 07:59:57.702685 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session 713b76caded84f269397165703739870-64eb15b0505a4649a0043be00250c299 received from {username='slave'} at 127.0.99.136:47971
I20240503 07:59:57.702757 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session 713b76caded84f269397165703739870-64eb15b0505a4649a0043be00250c299 on tablet 64eb15b0505a4649a0043be00250c299 with peer 713b76caded84f269397165703739870
I20240503 07:59:57.705377 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-387f6b1606b94e11897372c0557870e4 received from {username='slave'} at 127.0.99.135:57621
I20240503 07:59:57.705457 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-387f6b1606b94e11897372c0557870e4 on tablet 387f6b1606b94e11897372c0557870e4 with peer cae9627a7600448faa8fcb7989b307ee
I20240503 07:59:57.735684 9215 tablet_copy_client.cc:523] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.737303 9215 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 07:59:57.752837 9240 tablet_copy_client.cc:523] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.753890 9240 tablet_bootstrap.cc:492] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 07:59:57.757037 9235 tablet_copy_client.cc:523] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.758280 9215 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.758342 9235 tablet_bootstrap.cc:492] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 07:59:57.758510 9215 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 07:59:57.758533 9245 tablet_copy_client.cc:523] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.758656 9215 ts_tablet_manager.cc:1367] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.002s
I20240503 07:59:57.759023 9215 raft_consensus.cc:348] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.759140 9215 raft_consensus.cc:729] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 07:59:57.759232 9215 consensus_queue.cc:260] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.759486 9215 ts_tablet_manager.cc:1398] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:57.759799 9245 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 07:59:57.760004 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-d74a03a296dc48e582faeddc7b978ccd received from {username='slave'} at 127.0.99.133:39381
I20240503 07:59:57.760071 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-d74a03a296dc48e582faeddc7b978ccd on tablet d74a03a296dc48e582faeddc7b978ccd with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 07:59:57.762799 8034 raft_consensus.cc:1203] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.768718 9250 tablet_copy_client.cc:523] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:57.770073 9250 tablet_bootstrap.cc:492] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 07:59:57.780851 9245 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.781095 9245 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 07:59:57.781280 9245 ts_tablet_manager.cc:1367] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.002s
I20240503 07:59:57.781651 9245 raft_consensus.cc:348] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.781754 9245 raft_consensus.cc:729] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 07:59:57.781867 9245 consensus_queue.cc:260] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:57.782189 9245 ts_tablet_manager.cc:1398] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:57.782603 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-8625e8b653624ea6acf6e04c3fd352a4 received from {username='slave'} at 127.0.99.134:35513
I20240503 07:59:57.782680 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-8625e8b653624ea6acf6e04c3fd352a4 on tablet 8625e8b653624ea6acf6e04c3fd352a4 with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 07:59:57.785331 8296 raft_consensus.cc:1203] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.789669 9235 tablet_bootstrap.cc:492] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.789925 9235 tablet_bootstrap.cc:492] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 07:59:57.790067 9235 ts_tablet_manager.cc:1367] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.010s
I20240503 07:59:57.790513 9235 raft_consensus.cc:348] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.790629 9235 raft_consensus.cc:729] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 07:59:57.790721 9235 consensus_queue.cc:260] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:57.791893 9235 ts_tablet_manager.cc:1398] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:57.791898 8428 raft_consensus.cc:1203] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.792376 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-3d8a5e7a21684a4782d43c5a1131f652 received from {username='slave'} at 127.0.99.135:56655
I20240503 07:59:57.792443 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-3d8a5e7a21684a4782d43c5a1131f652 on tablet 3d8a5e7a21684a4782d43c5a1131f652 with peer cae9627a7600448faa8fcb7989b307ee
I20240503 07:59:57.796983 9240 tablet_bootstrap.cc:492] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.797226 9240 tablet_bootstrap.cc:492] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 07:59:57.797361 9240 ts_tablet_manager.cc:1367] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.006s
I20240503 07:59:57.797698 9240 raft_consensus.cc:348] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.797788 9240 raft_consensus.cc:729] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 07:59:57.797873 9240 consensus_queue.cc:260] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:57.798118 9240 ts_tablet_manager.cc:1398] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:57.798928 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-37c8748a1c9d42b593b2381a1a870d2e received from {username='slave'} at 127.0.99.133:39381
I20240503 07:59:57.799005 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-37c8748a1c9d42b593b2381a1a870d2e on tablet 37c8748a1c9d42b593b2381a1a870d2e with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 07:59:57.800146 9250 tablet_bootstrap.cc:492] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:57.800388 9250 tablet_bootstrap.cc:492] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 07:59:57.800541 9250 ts_tablet_manager.cc:1367] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.000s
I20240503 07:59:57.800935 9250 raft_consensus.cc:348] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:57.801048 9250 raft_consensus.cc:729] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 07:59:57.801192 9250 consensus_queue.cc:260] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:57.801453 9250 ts_tablet_manager.cc:1398] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240503 07:59:57.801805 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 713b76caded84f269397165703739870-8a934439594245d79ed5980ff9c03277 received from {username='slave'} at 127.0.99.136:40651
I20240503 07:59:57.801878 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 713b76caded84f269397165703739870-8a934439594245d79ed5980ff9c03277 on tablet 8a934439594245d79ed5980ff9c03277 with peer 713b76caded84f269397165703739870
I20240503 07:59:57.820716 8034 raft_consensus.cc:1203] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.880308 8428 raft_consensus.cc:1203] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.892385 8165 raft_consensus.cc:1203] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.913859 8165 raft_consensus.cc:1203] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.935392 9889 raft_consensus.cc:1050] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 07:59:57.935925 9889 consensus_queue.cc:237] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 07:59:57.937268 6305 raft_consensus.cc:1261] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:57.937263 8428 raft_consensus.cc:1261] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:57.937564 6437 raft_consensus.cc:1261] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:57.937649 9891 consensus_queue.cc:1035] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:57.937831 9889 consensus_queue.cc:1035] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:57.937976 8623 consensus_queue.cc:1035] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:57.940650 8623 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:57.941375 6305 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:57.942695 6437 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:57.943284 6163 catalog_manager.cc:5484] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:57.944558 8428 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:57.947649 6699 consensus_queue.cc:237] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 07:59:57.948602 8428 raft_consensus.cc:1261] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:57.948704 6437 raft_consensus.cc:1261] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:57.948932 9891 consensus_queue.cc:1035] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:57.949117 8623 consensus_queue.cc:1035] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:57.950444 9808 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:57.950820 8428 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:57.950974 6437 raft_consensus.cc:2932] T 2fb6518dced248a88edde5cbbb25350d P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:57.951794 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2fb6518dced248a88edde5cbbb25350d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:57.953126 8165 raft_consensus.cc:1203] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.953591 6156 catalog_manager.cc:5484] T 2fb6518dced248a88edde5cbbb25350d P 713b76caded84f269397165703739870 reported cstate change: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:57.955667 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb6518dced248a88edde5cbbb25350d with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35598
I20240503 07:59:57.956053 9899 tablet_replica.cc:330] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 07:59:57.956444 9899 raft_consensus.cc:2229] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:57.956668 9899 raft_consensus.cc:2258] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:57.957679 9899 ts_tablet_manager.cc:1875] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:57.959386 9899 ts_tablet_manager.cc:1888] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:57.959456 9899 log.cc:1196] T 2fb6518dced248a88edde5cbbb25350d P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2fb6518dced248a88edde5cbbb25350d
I20240503 07:59:57.959897 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 2fb6518dced248a88edde5cbbb25350d (table rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]) successfully deleted
I20240503 07:59:57.961529 8296 raft_consensus.cc:1203] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:57.970492 8296 raft_consensus.cc:1203] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:58.002648 8034 raft_consensus.cc:1203] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:58.014581 9910 raft_consensus.cc:1050] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 07:59:58.015970 9910 consensus_queue.cc:237] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.017315 6699 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.017442 8296 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.017581 6305 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.017714 7884 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.017997 9917 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.018256 7884 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.020426 9917 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.021181 6305 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.021359 8296 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.023105 6699 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.023294 6164 catalog_manager.cc:5484] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.026398 6437 consensus_queue.cc:237] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.027513 8296 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.027909 6305 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.027890 9918 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.028486 9918 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.029374 9910 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.037478 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4925e30c2fb74934bbd5eb7415193c96 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.037591 8296 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.038077 6304 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.038386 6156 catalog_manager.cc:5484] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.041028 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet 4925e30c2fb74934bbd5eb7415193c96 with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45994
I20240503 07:59:58.041548 9937 tablet_replica.cc:330] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 07:59:58.041849 9937 raft_consensus.cc:2229] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.042181 9937 raft_consensus.cc:2258] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.043262 9937 ts_tablet_manager.cc:1875] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.045770 9937 ts_tablet_manager.cc:1888] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.045852 9937 log.cc:1196] T 4925e30c2fb74934bbd5eb7415193c96 P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/4925e30c2fb74934bbd5eb7415193c96
I20240503 07:59:58.046355 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet 4925e30c2fb74934bbd5eb7415193c96 (table rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]) successfully deleted
I20240503 07:59:58.067817 9483 raft_consensus.cc:1050] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 07:59:58.068308 9483 consensus_queue.cc:237] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.069664 8165 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.069706 6699 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.069980 6437 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.070101 9967 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.070502 9965 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.070683 9958 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.074422 9966 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.074847 6699 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.075110 6436 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.076454 8165 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.077643 6163 catalog_manager.cc:5484] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.081468 6568 consensus_queue.cc:237] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.082628 8165 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.082954 9958 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.083034 6699 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.083884 9958 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.084880 9967 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.085314 8165 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.086171 6699 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.086791 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd0045edfdb441f2ae6920b8a97555bb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.088093 6157 catalog_manager.cc:5484] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.090281 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet cd0045edfdb441f2ae6920b8a97555bb with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34676
I20240503 07:59:58.090776 9981 tablet_replica.cc:330] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 07:59:58.091082 9981 raft_consensus.cc:2229] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.091338 9981 raft_consensus.cc:2258] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.092499 9981 ts_tablet_manager.cc:1875] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.094508 8428 raft_consensus.cc:1203] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:58.094837 9981 ts_tablet_manager.cc:1888] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.094919 9981 log.cc:1196] T cd0045edfdb441f2ae6920b8a97555bb P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/cd0045edfdb441f2ae6920b8a97555bb
I20240503 07:59:58.095346 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet cd0045edfdb441f2ae6920b8a97555bb (table rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]) successfully deleted
I20240503 07:59:58.116698 8165 raft_consensus.cc:1203] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:58.128230 6568 consensus_queue.cc:237] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:58.129421 6436 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.129505 6304 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:58.129637 6502 consensus_peers.cc:485] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67b9602786f8422d962885dda25c6c85. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.129786 9965 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.129948 9967 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.131577 9958 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.131867 6436 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.134086 6156 catalog_manager.cc:5484] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.134276 6304 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.140837 6436 consensus_queue.cc:237] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:58.142246 6568 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.142331 6699 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.142602 9910 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:58.142817 6372 consensus_peers.cc:485] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d17b7dcf55364209bd26129488dd0184. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.143018 9910 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.144915 9918 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:58.145334 6568 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:58.146222 9808 raft_consensus.cc:1050] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 07:59:58.146580 9808 consensus_queue.cc:237] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.148272 6304 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.148492 6156 catalog_manager.cc:5484] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.148650 8034 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.149135 9891 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.149186 6568 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.149325 6699 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:58.150790 8623 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.151551 9889 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.154633 9889 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.156234 6568 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.156399 8034 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.156749 6156 catalog_manager.cc:5484] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.158179 6304 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.159991 6699 consensus_queue.cc:237] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.161293 6304 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.161657 9808 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.162874 8034 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.163136 9889 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.163275 9808 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.163758 6304 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.164198 8034 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.165096 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d7012d8274214cf58c70d52062474b3a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.165565 6163 catalog_manager.cc:5484] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.167903 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet d7012d8274214cf58c70d52062474b3a with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41936
I20240503 07:59:58.168534 10037 tablet_replica.cc:330] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 07:59:58.168973 10037 raft_consensus.cc:2229] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.169238 10037 raft_consensus.cc:2258] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.171041 10037 ts_tablet_manager.cc:1875] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.173197 10037 ts_tablet_manager.cc:1888] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.173280 10037 log.cc:1196] T d7012d8274214cf58c70d52062474b3a P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d7012d8274214cf58c70d52062474b3a
I20240503 07:59:58.173805 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet d7012d8274214cf58c70d52062474b3a (table rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]) successfully deleted
I20240503 07:59:58.206758 7884 raft_consensus.cc:1050] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 07:59:58.207193 7884 consensus_queue.cc:237] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.208972 8296 raft_consensus.cc:1261] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.209432 6699 raft_consensus.cc:1261] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.209506 6568 raft_consensus.cc:1261] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.209825 9917 consensus_queue.cc:1035] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.210005 9918 consensus_queue.cc:1035] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.210273 9918 consensus_queue.cc:1035] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.212881 9917 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.213595 6568 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.216554 6699 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.217128 6156 catalog_manager.cc:5484] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.217902 8296 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.222064 6436 consensus_queue.cc:237] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.223104 8296 raft_consensus.cc:1261] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.223163 6699 raft_consensus.cc:1261] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.223425 10083 consensus_queue.cc:1035] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.223731 9917 consensus_queue.cc:1035] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.225219 9917 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.225796 8296 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.226581 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82f6c17706cc4a40827501ecb7c3ebce with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.227891 6163 catalog_manager.cc:5484] T 82f6c17706cc4a40827501ecb7c3ebce P cae9627a7600448faa8fcb7989b307ee reported cstate change: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.229136 6699 raft_consensus.cc:2932] T 82f6c17706cc4a40827501ecb7c3ebce P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.230235 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f6c17706cc4a40827501ecb7c3ebce with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41936
I20240503 07:59:58.230515 10037 tablet_replica.cc:330] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 07:59:58.230634 10037 raft_consensus.cc:2229] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.230773 10037 raft_consensus.cc:2258] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.230974 9889 raft_consensus.cc:1050] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 07:59:58.231334 9889 consensus_queue.cc:237] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.232197 10037 ts_tablet_manager.cc:1875] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.232614 8034 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.232964 8623 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.233199 6436 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.233500 6568 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.233994 8623 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.234127 9808 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.234216 10037 ts_tablet_manager.cc:1888] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.234275 10037 log.cc:1196] T 82f6c17706cc4a40827501ecb7c3ebce P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/82f6c17706cc4a40827501ecb7c3ebce
I20240503 07:59:58.235374 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet 82f6c17706cc4a40827501ecb7c3ebce (table rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]) successfully deleted
I20240503 07:59:58.236593 8623 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.237437 6568 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.238327 6436 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.238451 8034 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.240490 6163 catalog_manager.cc:5484] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.243896 6699 consensus_queue.cc:237] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.244935 6436 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.244935 8034 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.245256 9808 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.245412 9889 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.246856 9808 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.247440 8034 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.248121 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8efd24415e484e2a9af88b3aef6f78cc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.249178 6164 catalog_manager.cc:5484] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.250288 6436 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.251381 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet 8efd24415e484e2a9af88b3aef6f78cc with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41936
I20240503 07:59:58.251580 10037 tablet_replica.cc:330] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 07:59:58.251722 10037 raft_consensus.cc:2229] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.251870 10037 raft_consensus.cc:2258] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.252657 10037 ts_tablet_manager.cc:1875] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.254563 10037 ts_tablet_manager.cc:1888] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.254629 10037 log.cc:1196] T 8efd24415e484e2a9af88b3aef6f78cc P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8efd24415e484e2a9af88b3aef6f78cc
I20240503 07:59:58.255295 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet 8efd24415e484e2a9af88b3aef6f78cc (table rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]) successfully deleted
I20240503 07:59:58.256669 10113 raft_consensus.cc:1050] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 07:59:58.257233 10113 consensus_queue.cc:237] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.258487 6436 raft_consensus.cc:1261] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.258960 9385 consensus_queue.cc:1035] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.259002 8296 raft_consensus.cc:1261] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.259294 6699 raft_consensus.cc:1261] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.259370 9385 consensus_queue.cc:1035] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.259766 9385 consensus_queue.cc:1035] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.262185 9385 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.262477 8296 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.262698 6699 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.265249 6436 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.265364 6156 catalog_manager.cc:5484] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.265655 10083 raft_consensus.cc:1050] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 07:59:58.265949 10083 consensus_queue.cc:237] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.267228 6699 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.267633 6304 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.267943 8034 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.267997 7884 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.268484 7884 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.268849 7884 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.270710 6304 consensus_queue.cc:237] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.271850 6436 raft_consensus.cc:1261] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.272171 8296 raft_consensus.cc:1261] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.272163 10053 consensus_queue.cc:1035] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.272253 9918 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.272498 10053 consensus_queue.cc:1035] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.272704 8034 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.274531 6304 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.274781 6164 catalog_manager.cc:5484] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.274794 9385 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.276327 8296 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.276409 6699 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.277253 6436 raft_consensus.cc:2932] T 957ae2f17c4c41d28287d5490c54572e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.278450 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 957ae2f17c4c41d28287d5490c54572e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.279206 6157 catalog_manager.cc:5484] T 957ae2f17c4c41d28287d5490c54572e P cae9627a7600448faa8fcb7989b307ee reported cstate change: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.280181 6436 consensus_queue.cc:237] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.281364 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet 957ae2f17c4c41d28287d5490c54572e with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45994
I20240503 07:59:58.281579 9937 tablet_replica.cc:330] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 07:59:58.281611 6304 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.281703 9937 raft_consensus.cc:2229] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.281828 9937 raft_consensus.cc:2258] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.281920 10083 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.282979 9937 ts_tablet_manager.cc:1875] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.283032 8034 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.283572 7884 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.283854 6304 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.284860 9937 ts_tablet_manager.cc:1888] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.284916 9937 log.cc:1196] T 957ae2f17c4c41d28287d5490c54572e P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/957ae2f17c4c41d28287d5490c54572e
I20240503 07:59:58.285113 9917 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.285780 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet 957ae2f17c4c41d28287d5490c54572e (table rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]) successfully deleted
I20240503 07:59:58.285981 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d74a03a296dc48e582faeddc7b978ccd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.286052 6157 catalog_manager.cc:5484] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.286916 8034 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.288022 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet d74a03a296dc48e582faeddc7b978ccd with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45994
I20240503 07:59:58.288384 9937 tablet_replica.cc:330] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 07:59:58.288513 9937 raft_consensus.cc:2229] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.288650 9937 raft_consensus.cc:2258] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.289947 9937 ts_tablet_manager.cc:1875] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.291337 9937 ts_tablet_manager.cc:1888] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.291395 9937 log.cc:1196] T d74a03a296dc48e582faeddc7b978ccd P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d74a03a296dc48e582faeddc7b978ccd
I20240503 07:59:58.291777 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet d74a03a296dc48e582faeddc7b978ccd (table rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]) successfully deleted
I20240503 07:59:58.302491 6304 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:58.303685 6698 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.303727 6568 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.304025 10132 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:58.304236 6242 consensus_peers.cc:485] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08511a1b1eab4da9b810283d12cccee3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.304580 10132 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.305599 10053 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:58.306944 6698 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:58.307682 6155 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.307796 6568 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:58.316025 6698 consensus_queue.cc:237] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:58.317227 6304 raft_consensus.cc:1261] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.317229 6568 raft_consensus.cc:1261] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:58.317433 6635 consensus_peers.cc:485] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4418f348094548daa128685753415583. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.317566 9808 consensus_queue.cc:1035] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.317735 9891 consensus_queue.cc:1035] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.319406 9891 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.319821 6568 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.321954 6157 catalog_manager.cc:5484] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.322619 10132 raft_consensus.cc:1050] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 07:59:58.322958 10132 consensus_queue.cc:237] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 07:59:58.324225 6436 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.324556 8428 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.324576 10053 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.325044 6568 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.321758 6304 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.326920 10113 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.328061 10113 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.330447 10132 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.332429 10083 raft_consensus.cc:1050] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 07:59:58.332742 6436 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.332816 10083 consensus_queue.cc:237] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.334147 8034 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.334506 9917 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.334880 6698 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.335186 6566 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.335170 6304 consensus_queue.cc:237] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:58.335847 9917 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.336145 9917 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.336755 6566 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:58.336979 6240 consensus_peers.cc:485] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e47575b08ad4f54b1f8d35db27d9e58. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.337090 6163 catalog_manager.cc:5484] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.337656 6436 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.337671 9385 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.338104 9385 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.338551 8428 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.342563 6568 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.344349 6304 consensus_queue.cc:237] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 07:59:58.345427 9917 raft_consensus.cc:1050] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 07:59:58.345758 9917 consensus_queue.cc:237] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 07:59:58.345950 9483 raft_consensus.cc:1050] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 07:59:58.346252 9483 consensus_queue.cc:237] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.346982 10113 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.347206 8428 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.347544 9385 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.347577 6698 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.347577 6437 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.347890 9965 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.348203 8165 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.348836 9965 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.348996 6567 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.350345 6568 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.351615 10053 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.352067 9918 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.352371 6698 raft_consensus.cc:1261] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.352936 6567 raft_consensus.cc:1261] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.353232 8427 raft_consensus.cc:1261] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.353209 9910 consensus_queue.cc:1035] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.353196 6566 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.353701 6698 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.354847 9965 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.354790 8034 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.354975 9910 consensus_queue.cc:1035] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.355664 6163 catalog_manager.cc:5484] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.355919 9910 consensus_queue.cc:1035] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.356079 6155 catalog_manager.cc:5484] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.358732 9385 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.358808 6436 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.360553 8428 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.361538 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e03d7158b64f434a8871f009864a0b72 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.361974 9967 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.363270 6157 catalog_manager.cc:5484] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 reported cstate change: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.364287 6698 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.364343 8165 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.366114 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet e03d7158b64f434a8871f009864a0b72 with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34676
I20240503 07:59:58.366317 9981 tablet_replica.cc:330] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 07:59:58.366446 9981 raft_consensus.cc:2229] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.366586 9981 raft_consensus.cc:2258] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.367063 6157 catalog_manager.cc:5484] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.367300 9917 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.368940 9981 ts_tablet_manager.cc:1875] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.368955 6566 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.372673 8427 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.374946 6437 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.376224 6157 catalog_manager.cc:5484] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870 reported cstate change: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.376703 9981 ts_tablet_manager.cc:1888] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.376775 9981 log.cc:1196] T e03d7158b64f434a8871f009864a0b72 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e03d7158b64f434a8871f009864a0b72
I20240503 07:59:58.377889 6568 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.380079 6699 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.381186 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet e03d7158b64f434a8871f009864a0b72 (table rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]) successfully deleted
I20240503 07:59:58.384321 6437 consensus_queue.cc:237] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 07:59:58.385759 6568 consensus_queue.cc:237] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.385833 8427 raft_consensus.cc:1261] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.386162 10083 consensus_queue.cc:1035] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.386992 8165 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.387343 6699 raft_consensus.cc:1261] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.387678 10083 consensus_queue.cc:1035] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.387967 9958 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.388312 6436 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.388656 9958 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.389596 10083 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.390393 6699 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.390805 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a934439594245d79ed5980ff9c03277 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.391083 8427 raft_consensus.cc:2932] T 8a934439594245d79ed5980ff9c03277 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.391268 9483 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.392560 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8625e8b653624ea6acf6e04c3fd352a4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.392877 8165 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.393533 6156 catalog_manager.cc:5484] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.393831 6164 catalog_manager.cc:5484] T 8a934439594245d79ed5980ff9c03277 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.395908 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet 8625e8b653624ea6acf6e04c3fd352a4 with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45994
I20240503 07:59:58.395831 6437 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.396109 9937 tablet_replica.cc:330] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 07:59:58.396242 9937 raft_consensus.cc:2229] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.396423 9937 raft_consensus.cc:2258] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.397145 9937 ts_tablet_manager.cc:1875] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.397265 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a934439594245d79ed5980ff9c03277 with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41936
I20240503 07:59:58.399153 9937 ts_tablet_manager.cc:1888] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.399231 9937 log.cc:1196] T 8625e8b653624ea6acf6e04c3fd352a4 P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8625e8b653624ea6acf6e04c3fd352a4
I20240503 07:59:58.399665 10037 tablet_replica.cc:330] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 07:59:58.399700 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet 8625e8b653624ea6acf6e04c3fd352a4 (table rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]) successfully deleted
I20240503 07:59:58.399802 10037 raft_consensus.cc:2229] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.399921 10037 raft_consensus.cc:2258] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.400511 10037 ts_tablet_manager.cc:1875] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.402616 10037 ts_tablet_manager.cc:1888] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.402673 10037 log.cc:1196] T 8a934439594245d79ed5980ff9c03277 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8a934439594245d79ed5980ff9c03277
I20240503 07:59:58.403327 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet 8a934439594245d79ed5980ff9c03277 (table rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]) successfully deleted
I20240503 07:59:58.472548 6437 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e"
dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54090
I20240503 07:59:58.472692 6437 raft_consensus.cc:595] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:58.482017 9918 raft_consensus.cc:1050] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 07:59:58.482359 9918 consensus_queue.cc:237] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.483611 6568 raft_consensus.cc:1261] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.483611 6304 raft_consensus.cc:1261] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.483660 8165 raft_consensus.cc:1261] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.483940 9917 consensus_queue.cc:1035] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.484141 9918 consensus_queue.cc:1035] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.484412 7884 consensus_queue.cc:1035] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.486456 10288 raft_consensus.cc:1050] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 07:59:58.486809 10288 consensus_queue.cc:237] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.487135 10083 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.487994 6304 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.488687 8164 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.489219 8165 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.489755 6699 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.490406 6304 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.490411 6568 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.490960 6164 catalog_manager.cc:5484] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.491951 9917 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.492086 10083 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.492502 9918 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.494299 9918 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.495194 6437 consensus_queue.cc:237] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.495870 6304 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.497706 8165 raft_consensus.cc:1261] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.497663 6699 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.498270 6566 raft_consensus.cc:1261] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.498349 10083 consensus_queue.cc:1035] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.499015 9917 consensus_queue.cc:1035] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.499866 6157 catalog_manager.cc:5484] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.499859 8164 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.502452 9918 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.502988 8164 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.503651 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a1dab998bb84cc9880252383cb1e982 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.503747 6437 consensus_queue.cc:237] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.504446 6568 raft_consensus.cc:2932] T 9a1dab998bb84cc9880252383cb1e982 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.504932 6304 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.505470 6164 catalog_manager.cc:5484] T 9a1dab998bb84cc9880252383cb1e982 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.505973 8164 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.506320 9918 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.506950 9918 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.507701 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a1dab998bb84cc9880252383cb1e982 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35598
I20240503 07:59:58.508646 10288 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.509091 6304 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.509979 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9e91e1b70504ac695eae61f72d3c149 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.511063 6157 catalog_manager.cc:5484] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.511188 6568 consensus_queue.cc:237] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:58.512781 6699 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:58.513109 6504 consensus_peers.cc:485] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 885b10f956b54ebaa82614f8a1f979d1. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.513376 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e91e1b70504ac695eae61f72d3c149 with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45994
I20240503 07:59:58.513505 9966 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.513619 9937 tablet_replica.cc:330] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 07:59:58.513746 9937 raft_consensus.cc:2229] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.513865 9937 raft_consensus.cc:2258] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.513918 6304 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.514251 9966 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.514840 9937 ts_tablet_manager.cc:1875] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.515767 8164 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.515815 9965 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.516744 9937 ts_tablet_manager.cc:1888] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.516809 9937 log.cc:1196] T d9e91e1b70504ac695eae61f72d3c149 P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d9e91e1b70504ac695eae61f72d3c149
I20240503 07:59:58.517843 6304 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.518395 6164 catalog_manager.cc:5484] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.518716 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet d9e91e1b70504ac695eae61f72d3c149 (table rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]) successfully deleted
I20240503 07:59:58.521822 6699 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.521996 10301 tablet_replica.cc:330] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 07:59:58.522104 10301 raft_consensus.cc:2229] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.522250 10301 raft_consensus.cc:2258] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.523317 10301 ts_tablet_manager.cc:1875] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.525005 10301 ts_tablet_manager.cc:1888] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.525089 10301 log.cc:1196] T 9a1dab998bb84cc9880252383cb1e982 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9a1dab998bb84cc9880252383cb1e982
I20240503 07:59:58.525465 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 9a1dab998bb84cc9880252383cb1e982 (table rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]) successfully deleted
I20240503 07:59:58.527153 6304 consensus_queue.cc:237] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 07:59:58.528337 10113 raft_consensus.cc:1050] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 07:59:58.528506 6699 raft_consensus.cc:1261] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:58.528625 6242 consensus_peers.cc:485] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa0b29181066483386cab4c6f921f8f0. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.528682 10113 consensus_queue.cc:237] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.528808 6568 raft_consensus.cc:1261] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.529285 10172 consensus_queue.cc:1035] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.529451 10173 consensus_queue.cc:1035] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.530418 8296 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.530488 6698 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.530802 6437 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.530810 10113 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.530954 10173 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.531220 10113 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.532241 9385 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:58.532905 6568 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:58.533993 6698 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 07:59:58.534035 10113 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.534716 6157 catalog_manager.cc:5484] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index -1 to 4, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.535709 8296 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.535961 6698 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.538596 6163 catalog_manager.cc:5484] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee reported cstate change: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.539099 9966 raft_consensus.cc:1050] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 07:59:58.539438 9966 consensus_queue.cc:237] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.540251 6437 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.540714 6436 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.542109 9967 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.544019 6304 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.544508 8034 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.545588 9967 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.548770 9967 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.549197 6304 consensus_queue.cc:237] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.550031 9967 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.550220 6698 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.550225 8296 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.550642 10113 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.550742 6305 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.550805 9385 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.551707 8034 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:58.552641 6568 consensus_queue.cc:237] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:58.552641 6156 catalog_manager.cc:5484] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.554008 6699 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.554396 6305 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.554790 9966 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.554970 9483 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.555171 6437 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
W20240503 07:59:58.555595 6504 consensus_peers.cc:485] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d608390b027432f8a3f89130c051759. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.556986 9958 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.557410 6305 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.557843 9385 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.558305 6699 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.559093 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 387f6b1606b94e11897372c0557870e4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.559727 6699 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.559795 8296 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.560840 6164 catalog_manager.cc:5484] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:58.561041 6164 catalog_manager.cc:5484] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.564801 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet 387f6b1606b94e11897372c0557870e4 with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34676
I20240503 07:59:58.564981 9981 tablet_replica.cc:330] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 07:59:58.565124 9981 raft_consensus.cc:2229] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.565271 9981 raft_consensus.cc:2258] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.565800 9981 ts_tablet_manager.cc:1875] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.567423 9981 ts_tablet_manager.cc:1888] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.567468 9981 log.cc:1196] T 387f6b1606b94e11897372c0557870e4 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/387f6b1606b94e11897372c0557870e4
I20240503 07:59:58.567837 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet 387f6b1606b94e11897372c0557870e4 (table rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]) successfully deleted
I20240503 07:59:58.572233 6305 consensus_queue.cc:237] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 07:59:58.573745 6437 raft_consensus.cc:1261] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.573748 6568 raft_consensus.cc:1261] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.574115 9385 consensus_queue.cc:1035] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.574443 9385 consensus_queue.cc:1035] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.576607 10113 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:58.577018 6568 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:58.578727 6436 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 07:59:58.579352 6163 catalog_manager.cc:5484] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
W20240503 07:59:58.580371 6241 consensus_peers.cc:485] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffd6a5145bf64148b1c9db148905f92b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.589720 9483 raft_consensus.cc:1050] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 07:59:58.590037 9483 consensus_queue.cc:237] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.591173 8296 raft_consensus.cc:1261] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.591499 6305 raft_consensus.cc:1261] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.591511 6436 raft_consensus.cc:1261] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.591545 9483 consensus_queue.cc:1035] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.591840 9483 consensus_queue.cc:1035] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.591991 9966 consensus_queue.cc:1035] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.593828 9483 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.595175 6305 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.595994 6436 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.596652 8296 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.597039 6156 catalog_manager.cc:5484] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.600445 6568 consensus_queue.cc:237] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 07:59:58.601526 6305 raft_consensus.cc:1261] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.601832 9965 consensus_queue.cc:1035] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.601924 8296 raft_consensus.cc:1261] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.602298 9965 consensus_queue.cc:1035] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.603675 9965 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.604017 6304 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.604202 8296 raft_consensus.cc:2932] T 3d8a5e7a21684a4782d43c5a1131f652 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 07:59:58.605814 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d8a5e7a21684a4782d43c5a1131f652 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.606185 6164 catalog_manager.cc:5484] T 3d8a5e7a21684a4782d43c5a1131f652 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 5 to 6, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.606449 8623 raft_consensus.cc:1050] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 07:59:58.606822 8623 consensus_queue.cc:237] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 07:59:58.608139 6305 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.608211 8427 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.608319 6568 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.608675 9808 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.608973 9808 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.609357 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8a5e7a21684a4782d43c5a1131f652 with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34676
I20240503 07:59:58.609542 9981 tablet_replica.cc:330] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 07:59:58.609668 9981 raft_consensus.cc:2229] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.609802 9981 raft_consensus.cc:2258] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.609776 9889 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.610380 9981 ts_tablet_manager.cc:1875] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.611544 9889 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.611970 8427 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.612278 9981 ts_tablet_manager.cc:1888] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.612337 9981 log.cc:1196] T 3d8a5e7a21684a4782d43c5a1131f652 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3d8a5e7a21684a4782d43c5a1131f652
I20240503 07:59:58.613006 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet 3d8a5e7a21684a4782d43c5a1131f652 (table rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]) successfully deleted
I20240503 07:59:58.613005 6305 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.614974 6164 catalog_manager.cc:5484] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.615233 6568 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.618944 6699 consensus_queue.cc:237] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 07:59:58.619923 8427 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.620005 6568 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.620256 9889 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.620599 9889 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.621933 9808 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.622485 8427 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.622648 6568 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 07:59:58.623509 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64eb15b0505a4649a0043be00250c299 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.624325 6163 catalog_manager.cc:5484] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.626796 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 64eb15b0505a4649a0043be00250c299 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35598
I20240503 07:59:58.626966 10301 tablet_replica.cc:330] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 07:59:58.627100 10301 raft_consensus.cc:2229] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.627225 10301 raft_consensus.cc:2258] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.627908 10301 ts_tablet_manager.cc:1875] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.629684 10301 ts_tablet_manager.cc:1888] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.629760 10301 log.cc:1196] T 64eb15b0505a4649a0043be00250c299 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/64eb15b0505a4649a0043be00250c299
I20240503 07:59:58.630257 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 64eb15b0505a4649a0043be00250c299 (table rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]) successfully deleted
I20240503 07:59:58.647619 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58042
I20240503 07:59:58.647775 6568 raft_consensus.cc:595] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:58.660092 10412 ts_tablet_manager.cc:902] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:58.660573 10412 tablet_copy_client.cc:308] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 07:59:58.660996 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 67b9602786f8422d962885dda25c6c85 from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:56655)
I20240503 07:59:58.661114 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 67b9602786f8422d962885dda25c6c85 from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:56655: session id = cae9627a7600448faa8fcb7989b307ee-67b9602786f8422d962885dda25c6c85
I20240503 07:59:58.661742 6588 tablet_copy_source_session.cc:205] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:58.662542 10412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b9602786f8422d962885dda25c6c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:58.664443 10412 tablet_copy_client.cc:783] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:58.664574 10412 tablet_copy_client.cc:647] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:58.665550 10415 tablet_copy_client.cc:1002] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:58.680953 6437 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e"
dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54128
I20240503 07:59:58.681087 6437 raft_consensus.cc:595] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:58.698555 10452 ts_tablet_manager.cc:902] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 07:59:58.699002 10452 tablet_copy_client.cc:308] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 07:59:58.699625 6456 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet d17b7dcf55364209bd26129488dd0184 from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:54879)
I20240503 07:59:58.699719 6456 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet d17b7dcf55364209bd26129488dd0184 from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:54879: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-d17b7dcf55364209bd26129488dd0184
I20240503 07:59:58.700470 6456 tablet_copy_source_session.cc:205] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:58.701238 10452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b7dcf55364209bd26129488dd0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:58.702031 6699 consensus_queue.cc:237] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 07:59:58.703251 10452 tablet_copy_client.cc:783] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:58.703379 6437 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.703439 10452 tablet_copy_client.cc:647] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:58.703733 9808 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.703861 6305 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:58.704223 9808 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:58.705332 10456 tablet_copy_client.cc:1002] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:58.705587 8623 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.705994 6437 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.706578 6305 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 07:59:58.708496 6164 catalog_manager.cc:5484] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index -1 to 4, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
W20240503 07:59:58.711146 6633 consensus_peers.cc:485] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 551b0bab14bb4bb492885bada3e031a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:58.737038 9965 raft_consensus.cc:1050] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 07:59:58.737464 9965 consensus_queue.cc:237] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.739068 8164 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.739068 6437 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.739353 6305 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:58.739459 9958 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.739737 9966 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.740284 9966 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:58.741473 9917 raft_consensus.cc:979] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: : Instructing follower 34abd76ae3764e7f9641eaddaffdd5a5 to start an election
I20240503 07:59:58.741576 9917 raft_consensus.cc:1067] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Signalling peer 34abd76ae3764e7f9641eaddaffdd5a5 to start an election
I20240503 07:59:58.742235 6568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
from {username='slave'} at 127.0.99.130:35231
I20240503 07:59:58.742380 6568 raft_consensus.cc:482] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:58.742450 6568 raft_consensus.cc:3037] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:58.743475 6568 raft_consensus.cc:504] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.743932 6568 leader_election.cc:288] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 2 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285)
I20240503 07:59:58.744156 6436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:58.744258 6436 raft_consensus.cc:3032] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:58.744308 6436 raft_consensus.cc:729] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 07:59:58.744410 6699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 07:59:58.744458 6436 consensus_queue.cc:260] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.744503 6699 raft_consensus.cc:3037] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:58.744482 8034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c8748a1c9d42b593b2381a1a870d2e" candidate_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "61a9a0a13deb4d72b73f8623f6576766"
I20240503 07:59:58.744623 6436 raft_consensus.cc:3037] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:58.744663 8034 raft_consensus.cc:3037] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:58.745425 6699 raft_consensus.cc:2454] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 2.
I20240503 07:59:58.745627 6436 raft_consensus.cc:2454] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 2.
I20240503 07:59:58.745754 8034 raft_consensus.cc:2454] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34abd76ae3764e7f9641eaddaffdd5a5 in term 2.
I20240503 07:59:58.745806 9966 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.747274 6305 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.747917 6501 leader_election.cc:302] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5, 61a9a0a13deb4d72b73f8623f6576766; no voters:
I20240503 07:59:58.748056 9483 raft_consensus.cc:2781] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:58.747982 6163 catalog_manager.cc:5484] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.748133 9483 raft_consensus.cc:686] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 LEADER]: Becoming Leader. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:58.748263 9483 consensus_queue.cc:237] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.748627 8164 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.749689 6436 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.752198 6164 catalog_manager.cc:5484] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: term changed from 1 to 2, leader changed from 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) to 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:58.755069 6566 consensus_queue.cc:237] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 07:59:58.756063 6436 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.756063 8164 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:58.756368 9966 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.756528 9967 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:58.757946 9967 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.758332 6436 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.758491 8164 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 07:59:58.759374 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9023b7fb87a4adaa91995b883df8826 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:58.760190 6156 catalog_manager.cc:5484] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.762288 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet a9023b7fb87a4adaa91995b883df8826 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35598
I20240503 07:59:58.762481 10301 tablet_replica.cc:330] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 07:59:58.762607 10301 raft_consensus.cc:2229] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:58.762702 10301 raft_consensus.cc:2258] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:58.763242 10301 ts_tablet_manager.cc:1875] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:58.764925 10301 ts_tablet_manager.cc:1888] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:58.764984 10301 log.cc:1196] T a9023b7fb87a4adaa91995b883df8826 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a9023b7fb87a4adaa91995b883df8826
I20240503 07:59:58.765385 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet a9023b7fb87a4adaa91995b883df8826 (table rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]) successfully deleted
I20240503 07:59:58.772102 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58090
I20240503 07:59:58.772194 6568 raft_consensus.cc:595] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:58.806999 10556 ts_tablet_manager.cc:902] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:58.807477 10556 tablet_copy_client.cc:308] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 07:59:58.807974 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet 4418f348094548daa128685753415583 from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:54799)
I20240503 07:59:58.808079 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet 4418f348094548daa128685753415583 from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:54799: session id = 61a9a0a13deb4d72b73f8623f6576766-4418f348094548daa128685753415583
I20240503 07:59:58.809023 6719 tablet_copy_source_session.cc:205] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:58.809995 10556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4418f348094548daa128685753415583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:58.812022 10556 tablet_copy_client.cc:783] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:58.812157 10556 tablet_copy_client.cc:647] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:58.813302 10560 tablet_copy_client.cc:1002] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:58.818009 10571 ts_tablet_manager.cc:902] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:58.818784 10571 tablet_copy_client.cc:308] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 07:59:58.819221 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet 08511a1b1eab4da9b810283d12cccee3 from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:52377)
I20240503 07:59:58.819319 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet 08511a1b1eab4da9b810283d12cccee3 from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:52377: session id = 713b76caded84f269397165703739870-08511a1b1eab4da9b810283d12cccee3
I20240503 07:59:58.819952 6326 tablet_copy_source_session.cc:205] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:58.820681 10571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08511a1b1eab4da9b810283d12cccee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:58.822671 10571 tablet_copy_client.cc:783] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:58.822818 10571 tablet_copy_client.cc:647] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:58.823684 10575 tablet_copy_client.cc:1002] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:58.838029 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58152
I20240503 07:59:58.838138 6568 raft_consensus.cc:595] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:58.951458 10611 ts_tablet_manager.cc:902] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:58.951925 10611 tablet_copy_client.cc:308] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 07:59:58.952348 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet 9e47575b08ad4f54b1f8d35db27d9e58 from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:57621)
I20240503 07:59:58.952445 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet 9e47575b08ad4f54b1f8d35db27d9e58 from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:57621: session id = cae9627a7600448faa8fcb7989b307ee-9e47575b08ad4f54b1f8d35db27d9e58
I20240503 07:59:58.953150 6326 tablet_copy_source_session.cc:205] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:58.953891 10611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e47575b08ad4f54b1f8d35db27d9e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:58.955543 10611 tablet_copy_client.cc:783] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:58.955684 10611 tablet_copy_client.cc:647] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:58.956537 10615 tablet_copy_client.cc:1002] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:58.987107 9483 raft_consensus.cc:979] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: : Instructing follower 61a9a0a13deb4d72b73f8623f6576766 to start an election
I20240503 07:59:58.987216 9483 raft_consensus.cc:1067] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Signalling peer 61a9a0a13deb4d72b73f8623f6576766 to start an election
I20240503 07:59:58.987708 8034 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2"
dest_uuid: "61a9a0a13deb4d72b73f8623f6576766"
from {username='slave'} at 127.0.99.131:48679
I20240503 07:59:58.987866 8034 raft_consensus.cc:482] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:58.987931 8034 raft_consensus.cc:3037] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:58.988673 8034 raft_consensus.cc:504] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.989688 8034 leader_election.cc:288] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [CANDIDATE]: Term 2 election: Requested vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:58.989904 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2" candidate_uuid: "61a9a0a13deb4d72b73f8623f6576766" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 07:59:58.990015 6568 raft_consensus.cc:3032] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:58.990056 6568 raft_consensus.cc:729] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 07:59:58.990008 6436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2" candidate_uuid: "61a9a0a13deb4d72b73f8623f6576766" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 07:59:58.990139 6436 raft_consensus.cc:3037] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:58.990206 6568 consensus_queue.cc:260] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.990397 6568 raft_consensus.cc:3037] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:58.991006 6436 raft_consensus.cc:2454] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61a9a0a13deb4d72b73f8623f6576766 in term 2.
I20240503 07:59:58.991231 6568 raft_consensus.cc:2454] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61a9a0a13deb4d72b73f8623f6576766 in term 2.
I20240503 07:59:58.991519 7969 leader_election.cc:302] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5, 61a9a0a13deb4d72b73f8623f6576766; no voters:
I20240503 07:59:58.991703 9770 raft_consensus.cc:2781] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:58.991883 9770 raft_consensus.cc:686] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 LEADER]: Becoming Leader. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Running, Role: LEADER
I20240503 07:59:58.992168 9770 consensus_queue.cc:237] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:58.994081 6305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f40fae73f8f4c63b2d656ac880799e2" candidate_uuid: "61a9a0a13deb4d72b73f8623f6576766" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 07:59:58.994277 6305 raft_consensus.cc:3037] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:58.995005 6163 catalog_manager.cc:5484] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: term changed from 1 to 2, leader changed from 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) to 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "61a9a0a13deb4d72b73f8623f6576766" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:58.996529 6305 raft_consensus.cc:2454] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61a9a0a13deb4d72b73f8623f6576766 in term 2.
I20240503 07:59:59.015237 10618 ts_tablet_manager.cc:902] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:59.015650 10618 tablet_copy_client.cc:308] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 07:59:59.016029 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 7d608390b027432f8a3f89130c051759 from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:45731)
I20240503 07:59:59.016113 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 7d608390b027432f8a3f89130c051759 from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:45731: session id = 61a9a0a13deb4d72b73f8623f6576766-7d608390b027432f8a3f89130c051759
I20240503 07:59:59.016785 6588 tablet_copy_source_session.cc:205] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:59.017526 10618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d608390b027432f8a3f89130c051759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:59.019192 10618 tablet_copy_client.cc:783] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:59.019330 10618 tablet_copy_client.cc:647] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:59.020141 10622 tablet_copy_client.cc:1002] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:59.059176 10632 ts_tablet_manager.cc:902] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:59.060124 10632 tablet_copy_client.cc:308] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 07:59:59.060591 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet aa0b29181066483386cab4c6f921f8f0 from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:52377)
I20240503 07:59:59.060688 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet aa0b29181066483386cab4c6f921f8f0 from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:52377: session id = 713b76caded84f269397165703739870-aa0b29181066483386cab4c6f921f8f0
I20240503 07:59:59.061524 6326 tablet_copy_source_session.cc:205] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:59.062939 10632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa0b29181066483386cab4c6f921f8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:59.064987 10632 tablet_copy_client.cc:783] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:59.065217 10632 tablet_copy_client.cc:647] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:59.066126 10636 tablet_copy_client.cc:1002] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:59.074191 10648 ts_tablet_manager.cc:902] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:59.074707 10648 tablet_copy_client.cc:308] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 07:59:59.078493 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet ffd6a5145bf64148b1c9db148905f92b from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:54929)
I20240503 07:59:59.078583 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet ffd6a5145bf64148b1c9db148905f92b from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:54929: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-ffd6a5145bf64148b1c9db148905f92b
I20240503 07:59:59.079439 6326 tablet_copy_source_session.cc:205] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:59.080286 10648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd6a5145bf64148b1c9db148905f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:59.082355 10648 tablet_copy_client.cc:783] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:59.082499 10648 tablet_copy_client.cc:647] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:59.083410 10652 tablet_copy_client.cc:1002] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:59.095564 10675 ts_tablet_manager.cc:902] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 07:59:59.096043 10675 tablet_copy_client.cc:308] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 07:59:59.096593 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 885b10f956b54ebaa82614f8a1f979d1 from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:45731)
I20240503 07:59:59.096683 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 885b10f956b54ebaa82614f8a1f979d1 from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:45731: session id = 61a9a0a13deb4d72b73f8623f6576766-885b10f956b54ebaa82614f8a1f979d1
I20240503 07:59:59.097487 6588 tablet_copy_source_session.cc:205] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:59.098279 10675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b10f956b54ebaa82614f8a1f979d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:59.100425 10675 tablet_copy_client.cc:783] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:59.100603 10675 tablet_copy_client.cc:647] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:59.101666 10680 tablet_copy_client.cc:1002] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:59.164091 6436 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:59.164453 9483 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:59.226382 6699 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:59.226735 9483 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:59.227509 8034 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:59.227859 9483 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:59.234068 6568 consensus_queue.cc:237] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:59.235065 8034 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:59.235177 6699 raft_consensus.cc:1261] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:59.235400 9483 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:59.235569 9965 consensus_queue.cc:1035] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:59.236815 9483 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:59.237212 6699 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:59.237399 8034 raft_consensus.cc:2932] T 37c8748a1c9d42b593b2381a1a870d2e P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:59.239269 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37c8748a1c9d42b593b2381a1a870d2e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:59.239444 6163 catalog_manager.cc:5484] T 37c8748a1c9d42b593b2381a1a870d2e P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 5 to 7, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:59.241501 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c8748a1c9d42b593b2381a1a870d2e with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34676
I20240503 07:59:59.241797 10705 tablet_replica.cc:330] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 07:59:59.241955 10705 raft_consensus.cc:2229] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:59.242074 10705 raft_consensus.cc:2258] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:59.242628 10705 ts_tablet_manager.cc:1875] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:59.244350 10705 ts_tablet_manager.cc:1888] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:59:59.244423 10705 log.cc:1196] T 37c8748a1c9d42b593b2381a1a870d2e P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/37c8748a1c9d42b593b2381a1a870d2e
I20240503 07:59:59.244839 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet 37c8748a1c9d42b593b2381a1a870d2e (table rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]) successfully deleted
I20240503 07:59:59.301411 10707 ts_tablet_manager.cc:902] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 07:59:59.301903 10707 tablet_copy_client.cc:308] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 07:59:59.305208 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet 551b0bab14bb4bb492885bada3e031a0 from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:50045)
I20240503 07:59:59.305311 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet 551b0bab14bb4bb492885bada3e031a0 from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:50045: session id = cae9627a7600448faa8fcb7989b307ee-551b0bab14bb4bb492885bada3e031a0
I20240503 07:59:59.306128 6719 tablet_copy_source_session.cc:205] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:59.306820 10707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551b0bab14bb4bb492885bada3e031a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:59.308432 10707 tablet_copy_client.cc:783] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:59.308566 10707 tablet_copy_client.cc:647] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:59.309552 10712 tablet_copy_client.cc:1002] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 07:59:59.394150 6305 consensus_queue.cc:237] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 07:59:59.395509 6699 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:59.395567 6568 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:59.395877 10173 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:59.396221 9385 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:59.397490 10173 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:59.397940 6568 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
W20240503 07:59:59.399307 6242 consensus_peers.cc:485] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94f828890165466787015f21507e6ba2. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:59.400283 6164 catalog_manager.cc:5484] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:59.400715 6699 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 07:59:59.527362 6305 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:59.527806 9770 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:59.561213 6436 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:59.561739 9770 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:59.564924 6568 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:59.566154 10845 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:59.570837 8034 consensus_queue.cc:237] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 07:59:59.571775 6436 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:59.571784 6305 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:59.572085 10846 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:59.572224 10845 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:59.573565 10846 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:59.573945 6305 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:59.573973 6436 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:59.576117 6156 catalog_manager.cc:5484] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 5 to 7, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "61a9a0a13deb4d72b73f8623f6576766" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 07:59:59.576412 6151 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f40fae73f8f4c63b2d656ac880799e2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:59.578464 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f40fae73f8f4c63b2d656ac880799e2 with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41936
I20240503 07:59:59.578763 10851 tablet_replica.cc:330] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 07:59:59.578894 10851 raft_consensus.cc:2229] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:59.579001 10851 raft_consensus.cc:2258] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:59.579524 10851 ts_tablet_manager.cc:1875] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:59.581207 10851 ts_tablet_manager.cc:1888] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:59:59.581277 10851 log.cc:1196] T 2f40fae73f8f4c63b2d656ac880799e2 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2f40fae73f8f4c63b2d656ac880799e2
I20240503 07:59:59.581696 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet 2f40fae73f8f4c63b2d656ac880799e2 (table rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]) successfully deleted
I20240503 07:59:59.828665 10967 ts_tablet_manager.cc:902] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 07:59:59.829109 10967 tablet_copy_client.cc:308] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 07:59:59.829527 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet 94f828890165466787015f21507e6ba2 from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:42173)
I20240503 07:59:59.829659 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet 94f828890165466787015f21507e6ba2 from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:42173: session id = 61a9a0a13deb4d72b73f8623f6576766-94f828890165466787015f21507e6ba2
I20240503 07:59:59.830408 6326 tablet_copy_source_session.cc:205] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:59.831138 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f828890165466787015f21507e6ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:59.833002 10967 tablet_copy_client.cc:783] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:59.833170 10967 tablet_copy_client.cc:647] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:59.834059 10971 tablet_copy_client.cc:1002] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:00.165843 10412 tablet_copy_client.cc:523] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.167094 10412 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 08:00:00.183653 10412 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.183892 10412 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 08:00:00.184039 10412 ts_tablet_manager.cc:1367] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240503 08:00:00.184382 10412 raft_consensus.cc:348] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:00.184489 10412 raft_consensus.cc:729] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 08:00:00.184577 10412 consensus_queue.cc:260] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:00.184864 10412 ts_tablet_manager.cc:1398] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:00:00.185344 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-67b9602786f8422d962885dda25c6c85 received from {username='slave'} at 127.0.99.135:56655
I20240503 08:00:00.185415 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-67b9602786f8422d962885dda25c6c85 on tablet 67b9602786f8422d962885dda25c6c85 with peer cae9627a7600448faa8fcb7989b307ee
I20240503 08:00:00.205600 10452 tablet_copy_client.cc:523] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.206704 10452 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 08:00:00.210638 8296 raft_consensus.cc:1203] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:00.225401 10452 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.225618 10452 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 08:00:00.225764 10452 ts_tablet_manager.cc:1367] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20240503 08:00:00.226086 10452 raft_consensus.cc:348] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:00.226184 10452 raft_consensus.cc:729] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 08:00:00.226269 10452 consensus_queue.cc:260] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:00.226548 10452 ts_tablet_manager.cc:1398] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:00.227061 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-d17b7dcf55364209bd26129488dd0184 received from {username='slave'} at 127.0.99.134:54879
I20240503 08:00:00.227137 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-d17b7dcf55364209bd26129488dd0184 on tablet d17b7dcf55364209bd26129488dd0184 with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 08:00:00.313572 10556 tablet_copy_client.cc:523] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.314936 10556 tablet_bootstrap.cc:492] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 08:00:00.323956 10571 tablet_copy_client.cc:523] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.325301 10571 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 08:00:00.332401 10556 tablet_bootstrap.cc:492] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.332613 10556 tablet_bootstrap.cc:492] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 08:00:00.332762 10556 ts_tablet_manager.cc:1367] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240503 08:00:00.333120 10556 raft_consensus.cc:348] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:00.333220 10556 raft_consensus.cc:729] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 08:00:00.333343 10556 consensus_queue.cc:260] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:00.333631 10556 ts_tablet_manager.cc:1398] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:00:00.334090 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-4418f348094548daa128685753415583 received from {username='slave'} at 127.0.99.133:54799
I20240503 08:00:00.334193 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-4418f348094548daa128685753415583 on tablet 4418f348094548daa128685753415583 with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 08:00:00.344141 10571 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.344331 10571 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 08:00:00.344448 10571 ts_tablet_manager.cc:1367] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20240503 08:00:00.344769 10571 raft_consensus.cc:348] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:00.344859 10571 raft_consensus.cc:729] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 08:00:00.344942 10571 consensus_queue.cc:260] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:00.345227 10571 ts_tablet_manager.cc:1398] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:00:00.345661 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 713b76caded84f269397165703739870-08511a1b1eab4da9b810283d12cccee3 received from {username='slave'} at 127.0.99.136:52377
I20240503 08:00:00.345750 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 713b76caded84f269397165703739870-08511a1b1eab4da9b810283d12cccee3 on tablet 08511a1b1eab4da9b810283d12cccee3 with peer 713b76caded84f269397165703739870
I20240503 08:00:00.368247 8034 raft_consensus.cc:1203] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:00.406220 8427 raft_consensus.cc:1203] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:00.456823 10611 tablet_copy_client.cc:523] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.458081 8164 raft_consensus.cc:1203] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:00.458163 10611 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 08:00:00.476428 10611 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.476706 10611 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 08:00:00.476897 10611 ts_tablet_manager.cc:1367] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240503 08:00:00.477327 10611 raft_consensus.cc:348] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:00.477450 10611 raft_consensus.cc:729] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 08:00:00.477550 10611 consensus_queue.cc:260] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:00.477813 10611 ts_tablet_manager.cc:1398] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:00.478269 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-9e47575b08ad4f54b1f8d35db27d9e58 received from {username='slave'} at 127.0.99.135:57621
I20240503 08:00:00.478343 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-9e47575b08ad4f54b1f8d35db27d9e58 on tablet 9e47575b08ad4f54b1f8d35db27d9e58 with peer cae9627a7600448faa8fcb7989b307ee
I20240503 08:00:00.520486 10618 tablet_copy_client.cc:523] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.522257 10618 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 08:00:00.540067 10618 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.540288 10618 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 08:00:00.540418 10618 ts_tablet_manager.cc:1367] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240503 08:00:00.540817 10618 raft_consensus.cc:348] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:00.540938 10618 raft_consensus.cc:729] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 08:00:00.541029 10618 consensus_queue.cc:260] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:00.541410 10618 ts_tablet_manager.cc:1398] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:00:00.541890 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-7d608390b027432f8a3f89130c051759 received from {username='slave'} at 127.0.99.133:45731
I20240503 08:00:00.541966 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-7d608390b027432f8a3f89130c051759 on tablet 7d608390b027432f8a3f89130c051759 with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 08:00:00.566403 10632 tablet_copy_client.cc:523] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.567629 10632 tablet_bootstrap.cc:492] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 08:00:00.583695 10648 tablet_copy_client.cc:523] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.583748 10632 tablet_bootstrap.cc:492] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.583990 10632 tablet_bootstrap.cc:492] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 08:00:00.584133 10632 ts_tablet_manager.cc:1367] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240503 08:00:00.584472 10632 raft_consensus.cc:348] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:00.584570 10632 raft_consensus.cc:729] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 08:00:00.584659 10632 consensus_queue.cc:260] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:00.584945 10648 tablet_bootstrap.cc:492] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 08:00:00.585000 10632 ts_tablet_manager.cc:1398] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:00:00.585516 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 713b76caded84f269397165703739870-aa0b29181066483386cab4c6f921f8f0 received from {username='slave'} at 127.0.99.136:52377
I20240503 08:00:00.585600 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 713b76caded84f269397165703739870-aa0b29181066483386cab4c6f921f8f0 on tablet aa0b29181066483386cab4c6f921f8f0 with peer 713b76caded84f269397165703739870
I20240503 08:00:00.595750 8427 raft_consensus.cc:1203] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:00.601758 10648 tablet_bootstrap.cc:492] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.601933 10675 tablet_copy_client.cc:523] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.601979 10648 tablet_bootstrap.cc:492] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 08:00:00.602113 10648 ts_tablet_manager.cc:1367] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20240503 08:00:00.602437 10648 raft_consensus.cc:348] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:00.602538 10648 raft_consensus.cc:729] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 08:00:00.602614 10648 consensus_queue.cc:260] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:00.602844 10648 ts_tablet_manager.cc:1398] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:00.603111 10675 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 08:00:00.603261 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-ffd6a5145bf64148b1c9db148905f92b received from {username='slave'} at 127.0.99.134:54929
I20240503 08:00:00.603338 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-ffd6a5145bf64148b1c9db148905f92b on tablet ffd6a5145bf64148b1c9db148905f92b with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 08:00:00.619987 10675 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.620191 10675 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 08:00:00.620323 10675 ts_tablet_manager.cc:1367] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240503 08:00:00.620647 10675 raft_consensus.cc:348] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:00.620744 10675 raft_consensus.cc:729] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 08:00:00.620834 10675 consensus_queue.cc:260] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:00.621073 10675 ts_tablet_manager.cc:1398] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:00.621537 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-885b10f956b54ebaa82614f8a1f979d1 received from {username='slave'} at 127.0.99.133:45731
I20240503 08:00:00.621624 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-885b10f956b54ebaa82614f8a1f979d1 on tablet 885b10f956b54ebaa82614f8a1f979d1 with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 08:00:00.666468 8034 raft_consensus.cc:1203] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:00.731449 8034 raft_consensus.cc:1203] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:00.769483 11198 raft_consensus.cc:1050] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 08:00:00.769865 11198 consensus_queue.cc:237] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:00.771135 6305 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:00.771135 6436 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:00.771251 8296 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:00.771623 11152 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:00.771793 11205 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:00.771982 11198 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:00.774152 11205 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:00.774562 6305 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:00.776108 6436 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:00.776492 8296 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:00.777554 6164 catalog_manager.cc:5484] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:00.782769 6568 consensus_queue.cc:237] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:00.783859 8296 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:00.784196 6436 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:00.784482 11205 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:00.784767 11152 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:00.785943 9483 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:00.786465 8296 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:00.786701 6436 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:00.787811 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67b9602786f8422d962885dda25c6c85 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:00.788244 6157 catalog_manager.cc:5484] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:00.790165 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b9602786f8422d962885dda25c6c85 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35598
I20240503 08:00:00.790481 11218 tablet_replica.cc:330] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 08:00:00.790612 11218 raft_consensus.cc:2229] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:00.790758 11218 raft_consensus.cc:2258] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:00.791265 11218 ts_tablet_manager.cc:1875] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:00.792846 11218 ts_tablet_manager.cc:1888] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:00.792909 11218 log.cc:1196] T 67b9602786f8422d962885dda25c6c85 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/67b9602786f8422d962885dda25c6c85
I20240503 08:00:00.793341 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 67b9602786f8422d962885dda25c6c85 (table rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]) successfully deleted
I20240503 08:00:00.802417 9385 raft_consensus.cc:1050] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 08:00:00.802711 9385 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:00.803840 8427 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:00.803912 6568 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:00.804186 10734 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:00.804265 6699 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:00.804564 10734 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:00.805096 10734 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:00.806227 11220 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:00.806612 6699 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:00.807560 6568 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:00.808689 8427 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:00.808964 6163 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:00.809980 10707 tablet_copy_client.cc:523] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:00.811264 10707 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 08:00:00.812232 6305 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:00.813199 8427 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:00.813468 6699 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:00.813477 10173 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:00.813812 10173 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:00.814815 11220 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:00.815222 8427 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:00.815884 6699 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:00.816403 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08511a1b1eab4da9b810283d12cccee3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:00.818251 6157 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:00.820369 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet 08511a1b1eab4da9b810283d12cccee3 with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41936
I20240503 08:00:00.820812 11225 tablet_replica.cc:330] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 08:00:00.820935 11225 raft_consensus.cc:2229] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:00.821098 11225 raft_consensus.cc:2258] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:00.821588 11225 ts_tablet_manager.cc:1875] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:00.823127 11225 ts_tablet_manager.cc:1888] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:00.823186 11225 log.cc:1196] T 08511a1b1eab4da9b810283d12cccee3 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/08511a1b1eab4da9b810283d12cccee3
I20240503 08:00:00.823596 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet 08511a1b1eab4da9b810283d12cccee3 (table rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]) successfully deleted
I20240503 08:00:00.830622 10707 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:00.830848 10707 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 08:00:00.830996 10707 ts_tablet_manager.cc:1367] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240503 08:00:00.831378 10707 raft_consensus.cc:348] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:00.831476 10707 raft_consensus.cc:729] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 08:00:00.831569 10707 consensus_queue.cc:260] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:00.831831 10707 ts_tablet_manager.cc:1398] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:00.832293 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-551b0bab14bb4bb492885bada3e031a0 received from {username='slave'} at 127.0.99.135:50045
I20240503 08:00:00.832370 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-551b0bab14bb4bb492885bada3e031a0 on tablet 551b0bab14bb4bb492885bada3e031a0 with peer cae9627a7600448faa8fcb7989b307ee
I20240503 08:00:00.838223 8296 raft_consensus.cc:1203] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:00.978394 11226 raft_consensus.cc:1050] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 08:00:00.978911 11226 consensus_queue.cc:237] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:00.980324 6305 raft_consensus.cc:1261] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:00.980681 8034 raft_consensus.cc:1261] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:00.980711 6568 raft_consensus.cc:1261] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:00.980779 11228 consensus_queue.cc:1035] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:00.981079 11226 consensus_queue.cc:1035] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:00.981273 11228 consensus_queue.cc:1035] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:00.983006 10972 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:00.983525 6568 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:00.984439 8034 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:00.986279 6156 catalog_manager.cc:5484] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:00.986752 6305 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:00.990806 6699 consensus_queue.cc:237] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:00.992036 8034 raft_consensus.cc:1261] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:00.992036 6566 raft_consensus.cc:1261] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:00.992381 11228 consensus_queue.cc:1035] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:00.992558 11226 consensus_queue.cc:1035] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:00.994232 11228 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:00.994788 8034 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:00.995707 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4418f348094548daa128685753415583 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:00.996057 6566 raft_consensus.cc:2932] T 4418f348094548daa128685753415583 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:00.996773 6163 catalog_manager.cc:5484] T 4418f348094548daa128685753415583 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:00.999933 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 4418f348094548daa128685753415583 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35598
I20240503 08:00:01.001996 11218 tablet_replica.cc:330] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 08:00:01.002135 11218 raft_consensus.cc:2229] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:01.003240 11218 raft_consensus.cc:2258] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:01.003990 11218 ts_tablet_manager.cc:1875] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:01.005790 11218 ts_tablet_manager.cc:1888] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:01.005863 11218 log.cc:1196] T 4418f348094548daa128685753415583 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4418f348094548daa128685753415583
I20240503 08:00:01.006289 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 4418f348094548daa128685753415583 (table rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]) successfully deleted
I20240503 08:00:01.041283 10708 raft_consensus.cc:1050] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 08:00:01.041682 10708 consensus_queue.cc:237] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:01.043799 8164 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.043843 6699 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.044191 6566 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.044265 11277 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.044539 11280 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.044874 11277 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.047273 11277 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.047672 6566 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.047771 6699 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.049757 8164 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.050297 6156 catalog_manager.cc:5484] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.053815 6436 consensus_queue.cc:237] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:01.054869 6566 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.055186 8164 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.055349 11278 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.055819 11278 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.057108 10708 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.057687 8164 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.057688 6566 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.058493 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d17b7dcf55364209bd26129488dd0184 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:01.059724 8165 raft_consensus.cc:1203] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:01.059847 6157 catalog_manager.cc:5484] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.062165 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b7dcf55364209bd26129488dd0184 with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45994
I20240503 08:00:01.062489 11299 tablet_replica.cc:330] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 08:00:01.062659 11299 raft_consensus.cc:2229] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:01.062973 11299 raft_consensus.cc:2258] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:01.063915 11299 ts_tablet_manager.cc:1875] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:01.065917 11299 ts_tablet_manager.cc:1888] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:01.065984 11299 log.cc:1196] T d17b7dcf55364209bd26129488dd0184 P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d17b7dcf55364209bd26129488dd0184
I20240503 08:00:01.066433 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet d17b7dcf55364209bd26129488dd0184 (table rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]) successfully deleted
I20240503 08:00:01.170737 11198 raft_consensus.cc:1050] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 08:00:01.171162 11198 consensus_queue.cc:237] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:01.172444 8034 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.172449 6305 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.172478 6699 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.172827 11198 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.172983 11152 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.173204 9483 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.174829 11152 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.175390 8034 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.176590 6305 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.176636 6699 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.177564 6156 catalog_manager.cc:5484] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.180917 6566 consensus_queue.cc:237] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:01.182062 6305 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.182077 8034 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.182389 11152 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.182569 11198 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.184060 9483 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.184357 6305 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.185235 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d608390b027432f8a3f89130c051759 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:01.185487 8034 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.186389 6157 catalog_manager.cc:5484] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.188458 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d608390b027432f8a3f89130c051759 with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45994
I20240503 08:00:01.188647 11299 tablet_replica.cc:330] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 08:00:01.188776 11299 raft_consensus.cc:2229] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:01.188930 11299 raft_consensus.cc:2258] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:01.189725 11299 ts_tablet_manager.cc:1875] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:01.191550 11299 ts_tablet_manager.cc:1888] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:01.191560 10734 raft_consensus.cc:1050] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 08:00:01.191617 11299 log.cc:1196] T 7d608390b027432f8a3f89130c051759 P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d608390b027432f8a3f89130c051759
I20240503 08:00:01.191968 10734 consensus_queue.cc:237] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:01.192116 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet 7d608390b027432f8a3f89130c051759 (table rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]) successfully deleted
I20240503 08:00:01.193145 6699 raft_consensus.cc:1261] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.193154 6566 raft_consensus.cc:1261] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.193192 8427 raft_consensus.cc:1261] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.193557 11220 consensus_queue.cc:1035] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.193711 10173 consensus_queue.cc:1035] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.193830 9385 consensus_queue.cc:1035] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.195982 11220 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:01.196413 6699 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:01.197715 8427 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:01.198731 6163 catalog_manager.cc:5484] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:01.199587 6566 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:01.202433 6305 consensus_queue.cc:237] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:01.203434 8427 raft_consensus.cc:1261] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.203776 10734 consensus_queue.cc:1035] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.205140 6699 raft_consensus.cc:1261] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.205441 10734 consensus_queue.cc:1035] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.206656 9385 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:01.208040 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa0b29181066483386cab4c6f921f8f0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:01.208123 6699 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:01.208534 6163 catalog_manager.cc:5484] T aa0b29181066483386cab4c6f921f8f0 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:01.210575 8427 raft_consensus.cc:2932] T aa0b29181066483386cab4c6f921f8f0 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:01.211927 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet aa0b29181066483386cab4c6f921f8f0 with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41936
I20240503 08:00:01.212265 11225 tablet_replica.cc:330] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 08:00:01.212395 11225 raft_consensus.cc:2229] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:01.212536 11225 raft_consensus.cc:2258] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:01.213294 11225 ts_tablet_manager.cc:1875] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:01.214757 11225 ts_tablet_manager.cc:1888] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:01.214824 11225 log.cc:1196] T aa0b29181066483386cab4c6f921f8f0 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/aa0b29181066483386cab4c6f921f8f0
I20240503 08:00:01.215211 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet aa0b29181066483386cab4c6f921f8f0 (table rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]) successfully deleted
I20240503 08:00:01.334342 10967 tablet_copy_client.cc:523] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:01.335860 10967 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 08:00:01.356200 10967 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:01.356464 10967 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 08:00:01.356726 10967 ts_tablet_manager.cc:1367] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.002s
I20240503 08:00:01.357125 9483 raft_consensus.cc:1050] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 08:00:01.357439 10967 raft_consensus.cc:348] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:01.357576 10967 raft_consensus.cc:729] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 08:00:01.357506 9483 consensus_queue.cc:237] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:01.357666 10967 consensus_queue.cc:260] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:01.357916 10967 ts_tablet_manager.cc:1398] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:01.358835 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-94f828890165466787015f21507e6ba2 received from {username='slave'} at 127.0.99.133:42173
I20240503 08:00:01.358911 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-94f828890165466787015f21507e6ba2 on tablet 94f828890165466787015f21507e6ba2 with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 08:00:01.358884 6698 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.358928 8034 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.359355 6305 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.359371 11205 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.359879 11152 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.360183 11152 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.361913 9483 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.362250 8034 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.363492 6698 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.364037 6163 catalog_manager.cc:5484] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:01.366684 6305 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.368440 6566 consensus_queue.cc:237] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:01.369378 8034 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.369593 6305 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.369784 11152 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.370831 11152 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.371155 11152 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.371490 8034 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.373939 6157 catalog_manager.cc:5484] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.374567 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 885b10f956b54ebaa82614f8a1f979d1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:01.376477 6305 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:01.377162 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet 885b10f956b54ebaa82614f8a1f979d1 with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45994
I20240503 08:00:01.377358 11299 tablet_replica.cc:330] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 08:00:01.377496 11299 raft_consensus.cc:2229] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:01.377629 11299 raft_consensus.cc:2258] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:01.378574 11299 ts_tablet_manager.cc:1875] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:01.380394 11299 ts_tablet_manager.cc:1888] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:01.380455 11299 log.cc:1196] T 885b10f956b54ebaa82614f8a1f979d1 P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/885b10f956b54ebaa82614f8a1f979d1
I20240503 08:00:01.381011 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet 885b10f956b54ebaa82614f8a1f979d1 (table rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]) successfully deleted
I20240503 08:00:01.392302 8295 raft_consensus.cc:1203] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:01.392736 10173 raft_consensus.cc:1050] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 08:00:01.393097 10173 consensus_queue.cc:237] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:01.394352 6436 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.394701 10173 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.395143 8295 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.395536 6566 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.395502 10173 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.395964 10173 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.397727 11220 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.398118 6566 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.400207 6436 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.400329 6157 catalog_manager.cc:5484] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.400463 8295 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.404501 6305 consensus_queue.cc:237] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:01.405503 8295 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.405576 6436 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.405891 10173 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.406046 10734 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.407869 10173 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.408427 8295 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.409237 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e47575b08ad4f54b1f8d35db27d9e58 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:01.409920 6156 catalog_manager.cc:5484] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:01.410792 6436 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.412292 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e47575b08ad4f54b1f8d35db27d9e58 with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41936
I20240503 08:00:01.412482 11225 tablet_replica.cc:330] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 08:00:01.412616 11225 raft_consensus.cc:2229] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:01.412744 11225 raft_consensus.cc:2258] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:01.413446 11225 ts_tablet_manager.cc:1875] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:01.415071 11225 ts_tablet_manager.cc:1888] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:01.415138 11225 log.cc:1196] T 9e47575b08ad4f54b1f8d35db27d9e58 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9e47575b08ad4f54b1f8d35db27d9e58
I20240503 08:00:01.415737 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet 9e47575b08ad4f54b1f8d35db27d9e58 (table rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]) successfully deleted
I20240503 08:00:01.495812 8034 raft_consensus.cc:1203] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:01.520640 9385 raft_consensus.cc:1050] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 08:00:01.520998 9385 consensus_queue.cc:237] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:01.522289 8164 raft_consensus.cc:1261] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.522289 6436 raft_consensus.cc:1261] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.522354 6566 raft_consensus.cc:1261] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.522635 10173 consensus_queue.cc:1035] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.522783 10734 consensus_queue.cc:1035] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.522927 11220 consensus_queue.cc:1035] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.524499 10173 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.524884 6566 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.525235 6436 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.527524 6156 catalog_manager.cc:5484] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.527719 8164 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.531147 6305 consensus_queue.cc:237] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:01.532083 6436 raft_consensus.cc:1261] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.532114 8164 raft_consensus.cc:1261] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.532411 11220 consensus_queue.cc:1035] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.532727 10734 consensus_queue.cc:1035] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.534091 10734 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.534469 8164 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.535552 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ffd6a5145bf64148b1c9db148905f92b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:01.536104 6163 catalog_manager.cc:5484] T ffd6a5145bf64148b1c9db148905f92b P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:01.536916 6436 raft_consensus.cc:2932] T ffd6a5145bf64148b1c9db148905f92b P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:01.538411 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd6a5145bf64148b1c9db148905f92b with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41936
I20240503 08:00:01.538637 11225 tablet_replica.cc:330] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 08:00:01.538765 11225 raft_consensus.cc:2229] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:01.538916 11225 raft_consensus.cc:2258] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:01.539561 11225 ts_tablet_manager.cc:1875] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:01.541275 11225 ts_tablet_manager.cc:1888] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:01.541344 11225 log.cc:1196] T ffd6a5145bf64148b1c9db148905f92b P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ffd6a5145bf64148b1c9db148905f92b
I20240503 08:00:01.541769 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet ffd6a5145bf64148b1c9db148905f92b (table rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]) successfully deleted
I20240503 08:00:01.923971 11499 raft_consensus.cc:1050] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 08:00:01.924347 11499 consensus_queue.cc:237] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:01.925565 8295 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.925596 6436 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.925613 6305 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:01.926038 11499 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.926227 11501 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.926352 11498 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:01.928390 11498 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.928782 6305 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.928820 8295 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.930473 6436 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.931149 6163 catalog_manager.cc:5484] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 4 to 5, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:01.934334 6698 consensus_queue.cc:237] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:01.935462 8295 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.935511 6436 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:01.935864 11501 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.936024 11499 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:01.937362 11498 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.937865 6437 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.938419 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 551b0bab14bb4bb492885bada3e031a0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:01.938717 8295 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:01.940651 6157 catalog_manager.cc:5484] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 5 to 6, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:01.943109 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 551b0bab14bb4bb492885bada3e031a0 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35598
I20240503 08:00:01.943454 11506 tablet_replica.cc:330] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 08:00:01.943583 11506 raft_consensus.cc:2229] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:01.943729 11506 raft_consensus.cc:2258] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:01.944239 11506 ts_tablet_manager.cc:1875] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:01.946130 11506 ts_tablet_manager.cc:1888] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:01.946235 11506 log.cc:1196] T 551b0bab14bb4bb492885bada3e031a0 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/551b0bab14bb4bb492885bada3e031a0
I20240503 08:00:01.946653 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 551b0bab14bb4bb492885bada3e031a0 (table rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]) successfully deleted
I20240503 08:00:02.079064 10734 raft_consensus.cc:1050] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 08:00:02.079473 10734 consensus_queue.cc:237] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:02.081051 8034 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:02.081174 6568 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:02.081454 6698 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:02.081584 11517 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:02.081851 10734 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:02.082255 10734 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:02.083770 11515 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:02.084206 6698 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:02.085738 8034 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:02.086006 6568 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:02.087870 6156 catalog_manager.cc:5484] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 4 to 5, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:02.091452 6305 consensus_queue.cc:237] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:02.092674 6568 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:02.092746 8034 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:02.093046 11515 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:02.093858 11515 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:02.095157 11517 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:02.095590 8034 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:02.096024 6568 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:02.096652 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 94f828890165466787015f21507e6ba2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:02.097371 6163 catalog_manager.cc:5484] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 5 to 6, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:02.099438 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f828890165466787015f21507e6ba2 with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45994
I20240503 08:00:02.100024 11531 tablet_replica.cc:330] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 08:00:02.100205 11531 raft_consensus.cc:2229] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:02.100376 11531 raft_consensus.cc:2258] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:02.101040 11531 ts_tablet_manager.cc:1875] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:02.103104 11531 ts_tablet_manager.cc:1888] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:02.103174 11531 log.cc:1196] T 94f828890165466787015f21507e6ba2 P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/94f828890165466787015f21507e6ba2
I20240503 08:00:02.103610 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet 94f828890165466787015f21507e6ba2 (table rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]) successfully deleted
I20240503 08:00:02.165218 9316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:02.165693 6546 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:02.168797 8407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:02.176254 8014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:02.177448 6415 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:02.177668 8145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:02.187129 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:02.187963 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:02.205205 6284 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:00:02.250543 8034 consensus_queue.cc:237] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:02.251856 6305 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:02.251988 6437 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:00:02.252342 11585 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:02.252626 11584 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:02.254176 11585 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.254774 6437 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.257043 6155 catalog_manager.cc:5484] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: config changed from index 7 to 8, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 2 leader_uuid: "61a9a0a13deb4d72b73f8623f6576766" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:02.257191 6305 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
W20240503 08:00:02.257690 7968 consensus_peers.cc:485] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f40fae73f8f4c63b2d656ac880799e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.265190 6698 consensus_queue.cc:237] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:02.266355 6305 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.266422 8034 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.266697 11228 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:02.266892 6635 consensus_peers.cc:485] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7012d8274214cf58c70d52062474b3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.266893 11499 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.268481 11228 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.268926 8034 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.269228 6305 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.271263 6157 catalog_manager.cc:5484] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.278043 6568 consensus_queue.cc:237] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:02.279404 8295 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:02.279543 6503 consensus_peers.cc:485] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67b9602786f8422d962885dda25c6c85. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.279790 11604 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.279987 6437 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.280293 11604 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.281294 11603 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.281747 8295 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.283627 6437 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.283692 6155 catalog_manager.cc:5484] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:02.290278 6568 consensus_queue.cc:237] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:02.291461 8164 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.291465 6698 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:02.291723 6502 consensus_peers.cc:485] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd0045edfdb441f2ae6920b8a97555bb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.291885 11603 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.292045 9483 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.293678 11604 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 08:00:02.294153 6698 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 08:00:02.295681 8164 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 08:00:02.297626 6155 catalog_manager.cc:5484] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 6 to 7, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:02.305042 6437 consensus_queue.cc:237] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:02.306300 8164 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.306372 6568 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:02.306550 6373 consensus_peers.cc:485] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d17b7dcf55364209bd26129488dd0184. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.306671 11507 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.306850 11277 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.308408 11507 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 08:00:02.308827 8164 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 08:00:02.308907 6568 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 08:00:02.310997 6157 catalog_manager.cc:5484] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 6 to 7, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 08:00:02.319020 6305 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:02.320284 6698 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.320395 8427 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.320669 11517 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.320904 11515 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.322746 11515 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.323447 6698 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.324613 8427 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.325208 6163 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:02.327792 6240 consensus_peers.cc:485] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08511a1b1eab4da9b810283d12cccee3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.338419 6305 consensus_queue.cc:237] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:02.339615 8034 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:02.339622 6242 consensus_peers.cc:485] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94f828890165466787015f21507e6ba2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.339699 6568 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.339956 11517 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.340119 10734 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.341557 11513 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.341987 8034 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.343547 6568 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.344338 6163 catalog_manager.cc:5484] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.352506 6568 consensus_queue.cc:237] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:02.353967 8164 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.354135 6437 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.354420 11603 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.354573 11604 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.356987 11603 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.357403 8164 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.358733 6437 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
W20240503 08:00:02.360275 6502 consensus_peers.cc:485] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8625e8b653624ea6acf6e04c3fd352a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.361508 6157 catalog_manager.cc:5484] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.369294 6305 consensus_queue.cc:237] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:02.370565 8295 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.370625 6437 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:02.370922 6241 consensus_peers.cc:485] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e47575b08ad4f54b1f8d35db27d9e58. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.370922 11517 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.371147 11513 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.372769 11515 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.375063 8295 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.376612 6437 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.377288 6164 catalog_manager.cc:5484] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee reported cstate change: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.383278 6305 consensus_queue.cc:237] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:02.384508 8295 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.384506 6698 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.384864 10734 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.385030 11515 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:02.385387 6240 consensus_peers.cc:485] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 387f6b1606b94e11897372c0557870e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.386802 11513 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.387168 6698 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.387233 8295 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.389624 6155 catalog_manager.cc:5484] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:02.397292 6437 consensus_queue.cc:237] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:02.398584 6305 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:02.398705 6372 consensus_peers.cc:485] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851): Couldn't send request to peer 66f9a693c3b3409dbcce8f3ee500bf5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d74a03a296dc48e582faeddc7b978ccd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.398766 8034 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.398900 11277 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.399120 11507 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.400485 11277 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.401046 8034 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.402649 6305 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } } }
I20240503 08:00:02.402824 6157 catalog_manager.cc:5484] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 6 to 7, NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:02.409622 6698 consensus_queue.cc:237] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:02.410882 6568 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.411063 8427 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.411242 11499 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:02.411424 6635 consensus_peers.cc:485] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 -> Peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): Couldn't send request to peer 61a9a0a13deb4d72b73f8623f6576766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64eb15b0505a4649a0043be00250c299. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.411609 11499 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.412796 11501 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 08:00:02.413671 8427 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 08:00:02.414484 6568 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } } }
I20240503 08:00:02.415985 6155 catalog_manager.cc:5484] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 6 to 7, NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:02.423594 6698 consensus_queue.cc:237] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:02.425048 8034 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.425153 6437 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.425401 11499 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:02.425575 6635 consensus_peers.cc:485] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8efd24415e484e2a9af88b3aef6f78cc. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.425837 11499 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.427119 11501 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.427485 8034 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.427822 6437 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.429828 6157 catalog_manager.cc:5484] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.436594 6568 consensus_queue.cc:237] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:02.437891 8034 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.437963 6305 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.438216 11603 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:02.438484 6502 consensus_peers.cc:485] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 885b10f956b54ebaa82614f8a1f979d1. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.438634 11604 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.439796 11603 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.440208 8034 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.440640 6305 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.442723 6164 catalog_manager.cc:5484] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:02.449961 6437 consensus_queue.cc:237] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:02.451222 8164 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:02.451503 6373 consensus_peers.cc:485] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 713b76caded84f269397165703739870 (127.0.99.136:38867): Couldn't send request to peer 713b76caded84f269397165703739870. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9e91e1b70504ac695eae61f72d3c149. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:02.451545 11277 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.451613 6305 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:02.451923 11277 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:02.453297 11507 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.453701 8164 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.454116 6305 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } } }
I20240503 08:00:02.455916 6157 catalog_manager.cc:5484] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 6 to 7, NON_VOTER 713b76caded84f269397165703739870 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:02.655989 11777 ts_tablet_manager.cc:902] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 08:00:02.656498 11777 tablet_copy_client.cc:308] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 08:00:02.656944 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet d7012d8274214cf58c70d52062474b3a from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:47971)
I20240503 08:00:02.657094 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet d7012d8274214cf58c70d52062474b3a from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:47971: session id = 713b76caded84f269397165703739870-d7012d8274214cf58c70d52062474b3a
I20240503 08:00:02.657853 6719 tablet_copy_source_session.cc:205] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.658664 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7012d8274214cf58c70d52062474b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.660378 11777 tablet_copy_client.cc:783] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.660507 11777 tablet_copy_client.cc:647] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.661497 11781 tablet_copy_client.cc:1002] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.682989 11782 ts_tablet_manager.cc:902] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 08:00:02.683472 11782 tablet_copy_client.cc:308] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 08:00:02.683964 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 67b9602786f8422d962885dda25c6c85 from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:35513)
I20240503 08:00:02.684067 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 67b9602786f8422d962885dda25c6c85 from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:35513: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-67b9602786f8422d962885dda25c6c85
I20240503 08:00:02.684762 6588 tablet_copy_source_session.cc:205] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.685585 11782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b9602786f8422d962885dda25c6c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.687594 11782 tablet_copy_client.cc:783] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.687726 11782 tablet_copy_client.cc:647] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.688648 11786 tablet_copy_client.cc:1002] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.715138 11787 ts_tablet_manager.cc:902] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 08:00:02.715692 11787 tablet_copy_client.cc:308] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 08:00:02.716169 6456 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet d17b7dcf55364209bd26129488dd0184 from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:39381)
I20240503 08:00:02.716270 6456 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet d17b7dcf55364209bd26129488dd0184 from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:39381: session id = 61a9a0a13deb4d72b73f8623f6576766-d17b7dcf55364209bd26129488dd0184
I20240503 08:00:02.717027 6456 tablet_copy_source_session.cc:205] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.717813 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b7dcf55364209bd26129488dd0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.719484 11787 tablet_copy_client.cc:783] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.719612 11787 tablet_copy_client.cc:647] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.720496 11791 tablet_copy_client.cc:1002] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.758699 11817 ts_tablet_manager.cc:902] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 08:00:02.759366 11817 tablet_copy_client.cc:308] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 08:00:02.763131 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 8625e8b653624ea6acf6e04c3fd352a4 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:42151)
I20240503 08:00:02.763228 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 8625e8b653624ea6acf6e04c3fd352a4 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:42151: session id = 0394d429c78744aca1220af8f39b11c1-8625e8b653624ea6acf6e04c3fd352a4
I20240503 08:00:02.764078 6588 tablet_copy_source_session.cc:205] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.766850 11817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8625e8b653624ea6acf6e04c3fd352a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.771968 11817 tablet_copy_client.cc:783] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.772147 11817 tablet_copy_client.cc:647] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.773488 11821 tablet_copy_client.cc:1002] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.847100 11873 ts_tablet_manager.cc:902] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285)
I20240503 08:00:02.847548 11873 tablet_copy_client.cc:308] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:41285
I20240503 08:00:02.850639 8054 tablet_copy_service.cc:140] P 61a9a0a13deb4d72b73f8623f6576766: Received BeginTabletCopySession request for tablet 2f40fae73f8f4c63b2d656ac880799e2 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:39435)
I20240503 08:00:02.850749 8054 tablet_copy_service.cc:161] P 61a9a0a13deb4d72b73f8623f6576766: Beginning new tablet copy session on tablet 2f40fae73f8f4c63b2d656ac880799e2 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:39435: session id = 0394d429c78744aca1220af8f39b11c1-2f40fae73f8f4c63b2d656ac880799e2
I20240503 08:00:02.851768 8054 tablet_copy_source_session.cc:205] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.852717 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f40fae73f8f4c63b2d656ac880799e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.854452 11873 tablet_copy_client.cc:783] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.854593 11873 tablet_copy_client.cc:647] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.855645 11877 tablet_copy_client.cc:1002] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.874080 11878 ts_tablet_manager.cc:902] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 08:00:02.874608 11878 tablet_copy_client.cc:308] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 08:00:02.875053 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet 9e47575b08ad4f54b1f8d35db27d9e58 from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:54929)
I20240503 08:00:02.875140 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet 9e47575b08ad4f54b1f8d35db27d9e58 from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:54929: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-9e47575b08ad4f54b1f8d35db27d9e58
I20240503 08:00:02.875815 6326 tablet_copy_source_session.cc:205] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.876711 11883 ts_tablet_manager.cc:902] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 08:00:02.876749 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e47575b08ad4f54b1f8d35db27d9e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.877302 11883 tablet_copy_client.cc:308] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 08:00:02.877753 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet cd0045edfdb441f2ae6920b8a97555bb from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:56655)
I20240503 08:00:02.877852 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet cd0045edfdb441f2ae6920b8a97555bb from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:56655: session id = cae9627a7600448faa8fcb7989b307ee-cd0045edfdb441f2ae6920b8a97555bb
I20240503 08:00:02.878660 6588 tablet_copy_source_session.cc:205] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.878727 11878 tablet_copy_client.cc:783] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.878856 11878 tablet_copy_client.cc:647] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.879891 11882 tablet_copy_client.cc:1002] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.880024 11883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd0045edfdb441f2ae6920b8a97555bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.882104 11883 tablet_copy_client.cc:783] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.882251 11883 tablet_copy_client.cc:647] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.883195 11886 tablet_copy_client.cc:1002] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.899935 11888 ts_tablet_manager.cc:902] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 08:00:02.900424 11888 tablet_copy_client.cc:308] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 08:00:02.900919 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet 8efd24415e484e2a9af88b3aef6f78cc from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:47971)
I20240503 08:00:02.901026 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet 8efd24415e484e2a9af88b3aef6f78cc from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:47971: session id = 713b76caded84f269397165703739870-8efd24415e484e2a9af88b3aef6f78cc
I20240503 08:00:02.901763 6719 tablet_copy_source_session.cc:205] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.902541 11893 ts_tablet_manager.cc:902] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 08:00:02.902582 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efd24415e484e2a9af88b3aef6f78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.903061 11893 tablet_copy_client.cc:308] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 08:00:02.903530 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 885b10f956b54ebaa82614f8a1f979d1 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:42151)
I20240503 08:00:02.903625 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 885b10f956b54ebaa82614f8a1f979d1 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:42151: session id = 0394d429c78744aca1220af8f39b11c1-885b10f956b54ebaa82614f8a1f979d1
I20240503 08:00:02.904314 11888 tablet_copy_client.cc:783] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.904385 6588 tablet_copy_source_session.cc:205] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.904435 11888 tablet_copy_client.cc:647] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.905236 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b10f956b54ebaa82614f8a1f979d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.905421 11892 tablet_copy_client.cc:1002] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.906915 11893 tablet_copy_client.cc:783] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.907063 11893 tablet_copy_client.cc:647] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.908172 11897 tablet_copy_client.cc:1002] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.916208 11898 ts_tablet_manager.cc:902] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 08:00:02.916714 11898 tablet_copy_client.cc:308] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 08:00:02.916718 11901 ts_tablet_manager.cc:902] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 08:00:02.917230 11901 tablet_copy_client.cc:308] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 08:00:02.917243 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet 94f828890165466787015f21507e6ba2 from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:52377)
I20240503 08:00:02.917338 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet 94f828890165466787015f21507e6ba2 from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:52377: session id = 713b76caded84f269397165703739870-94f828890165466787015f21507e6ba2
I20240503 08:00:02.917649 6456 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet d74a03a296dc48e582faeddc7b978ccd from peer 66f9a693c3b3409dbcce8f3ee500bf5e ({username='slave'} at 127.0.99.134:54879)
I20240503 08:00:02.917739 6456 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet d74a03a296dc48e582faeddc7b978ccd from peer 66f9a693c3b3409dbcce8f3ee500bf5e at {username='slave'} at 127.0.99.134:54879: session id = 66f9a693c3b3409dbcce8f3ee500bf5e-d74a03a296dc48e582faeddc7b978ccd
I20240503 08:00:02.918092 6326 tablet_copy_source_session.cc:205] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.918414 6456 tablet_copy_source_session.cc:205] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.918816 11898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f828890165466787015f21507e6ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.919132 11901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74a03a296dc48e582faeddc7b978ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.920524 11898 tablet_copy_client.cc:783] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.920644 11898 tablet_copy_client.cc:647] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.920922 11901 tablet_copy_client.cc:783] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.921080 11901 tablet_copy_client.cc:647] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.921578 11902 tablet_copy_client.cc:1002] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.921965 11907 tablet_copy_client.cc:1002] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.927575 11908 ts_tablet_manager.cc:902] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 08:00:02.928089 11908 tablet_copy_client.cc:308] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 08:00:02.931463 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet 08511a1b1eab4da9b810283d12cccee3 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:38407)
I20240503 08:00:02.931543 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet 08511a1b1eab4da9b810283d12cccee3 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:38407: session id = 0394d429c78744aca1220af8f39b11c1-08511a1b1eab4da9b810283d12cccee3
I20240503 08:00:02.932137 6326 tablet_copy_source_session.cc:205] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.932920 11908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08511a1b1eab4da9b810283d12cccee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.934569 11908 tablet_copy_client.cc:783] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.934697 11908 tablet_copy_client.cc:647] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.935709 11912 tablet_copy_client.cc:1002] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.982031 11913 ts_tablet_manager.cc:902] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 08:00:02.982426 11913 tablet_copy_client.cc:308] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 08:00:02.982853 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet 64eb15b0505a4649a0043be00250c299 from peer 61a9a0a13deb4d72b73f8623f6576766 ({username='slave'} at 127.0.99.133:54799)
I20240503 08:00:02.982955 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet 64eb15b0505a4649a0043be00250c299 from peer 61a9a0a13deb4d72b73f8623f6576766 at {username='slave'} at 127.0.99.133:54799: session id = 61a9a0a13deb4d72b73f8623f6576766-64eb15b0505a4649a0043be00250c299
I20240503 08:00:02.983642 6719 tablet_copy_source_session.cc:205] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.984390 11913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64eb15b0505a4649a0043be00250c299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.986080 11913 tablet_copy_client.cc:783] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.986215 11913 tablet_copy_client.cc:647] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.987116 11917 tablet_copy_client.cc:1002] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:02.991264 11918 ts_tablet_manager.cc:902] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 08:00:02.991781 11918 tablet_copy_client.cc:308] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 08:00:02.992210 6456 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet d9e91e1b70504ac695eae61f72d3c149 from peer 713b76caded84f269397165703739870 ({username='slave'} at 127.0.99.136:40651)
I20240503 08:00:02.992301 6456 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet d9e91e1b70504ac695eae61f72d3c149 from peer 713b76caded84f269397165703739870 at {username='slave'} at 127.0.99.136:40651: session id = 713b76caded84f269397165703739870-d9e91e1b70504ac695eae61f72d3c149
I20240503 08:00:02.992949 6456 tablet_copy_source_session.cc:205] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:02.993682 11918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e91e1b70504ac695eae61f72d3c149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:02.995272 11918 tablet_copy_client.cc:783] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:02.995393 11918 tablet_copy_client.cc:647] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:02.996251 11922 tablet_copy_client.cc:1002] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:03.008992 11923 ts_tablet_manager.cc:902] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 08:00:03.009580 11923 tablet_copy_client.cc:308] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 08:00:03.010016 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet 387f6b1606b94e11897372c0557870e4 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:38407)
I20240503 08:00:03.010100 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet 387f6b1606b94e11897372c0557870e4 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:38407: session id = 0394d429c78744aca1220af8f39b11c1-387f6b1606b94e11897372c0557870e4
I20240503 08:00:03.010826 6326 tablet_copy_source_session.cc:205] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:03.011546 11923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387f6b1606b94e11897372c0557870e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:03.013181 11923 tablet_copy_client.cc:783] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:03.013303 11923 tablet_copy_client.cc:647] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:03.014139 11924 tablet_copy_client.cc:1002] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:04.161784 11777 tablet_copy_client.cc:523] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.163626 11777 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 08:00:04.185711 11777 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.185966 11777 tablet_bootstrap.cc:492] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 08:00:04.186116 11777 ts_tablet_manager.cc:1367] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.003s
I20240503 08:00:04.186483 11777 raft_consensus.cc:348] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:04.186616 11777 raft_consensus.cc:729] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 08:00:04.186719 11777 consensus_queue.cc:260] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:04.187022 11777 ts_tablet_manager.cc:1398] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:04.187435 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session 713b76caded84f269397165703739870-d7012d8274214cf58c70d52062474b3a received from {username='slave'} at 127.0.99.136:47971
I20240503 08:00:04.187502 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session 713b76caded84f269397165703739870-d7012d8274214cf58c70d52062474b3a on tablet d7012d8274214cf58c70d52062474b3a with peer 713b76caded84f269397165703739870
I20240503 08:00:04.188891 11782 tablet_copy_client.cc:523] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.190369 11782 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 08:00:04.214928 11782 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.215138 11782 tablet_bootstrap.cc:492] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 08:00:04.215273 11782 ts_tablet_manager.cc:1367] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.004s
I20240503 08:00:04.215677 11782 raft_consensus.cc:348] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:04.215776 11782 raft_consensus.cc:729] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 08:00:04.215849 11782 consensus_queue.cc:260] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:04.216126 11782 ts_tablet_manager.cc:1398] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:04.216619 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-67b9602786f8422d962885dda25c6c85 received from {username='slave'} at 127.0.99.134:35513
I20240503 08:00:04.216706 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-67b9602786f8422d962885dda25c6c85 on tablet 67b9602786f8422d962885dda25c6c85 with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 08:00:04.220744 11787 tablet_copy_client.cc:523] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.221980 11787 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 08:00:04.256307 11787 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.256597 11787 tablet_bootstrap.cc:492] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 08:00:04.256759 11787 ts_tablet_manager.cc:1367] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.004s
I20240503 08:00:04.257203 11787 raft_consensus.cc:348] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:04.257313 11787 raft_consensus.cc:729] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 08:00:04.257406 11787 consensus_queue.cc:260] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:04.257648 11787 ts_tablet_manager.cc:1398] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240503 08:00:04.258178 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-d17b7dcf55364209bd26129488dd0184 received from {username='slave'} at 127.0.99.133:39381
I20240503 08:00:04.258252 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-d17b7dcf55364209bd26129488dd0184 on tablet d17b7dcf55364209bd26129488dd0184 with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 08:00:04.273862 11817 tablet_copy_client.cc:523] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.275660 11817 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:04.280897 11817 log.cc:824] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:04.298789 11817 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.299052 11817 tablet_bootstrap.cc:492] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:04.299309 11817 ts_tablet_manager.cc:1367] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20240503 08:00:04.301040 11817 raft_consensus.cc:348] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.301266 11817 raft_consensus.cc:729] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:04.301520 11817 consensus_queue.cc:260] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.302227 11817 ts_tablet_manager.cc:1398] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.003s user 0.001s sys 0.004s
I20240503 08:00:04.302762 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-8625e8b653624ea6acf6e04c3fd352a4 received from {username='slave'} at 127.0.99.137:42151
I20240503 08:00:04.302834 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-8625e8b653624ea6acf6e04c3fd352a4 on tablet 8625e8b653624ea6acf6e04c3fd352a4 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:04.341251 9336 raft_consensus.cc:1203] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.355899 11873 tablet_copy_client.cc:523] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.357122 11873 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:04.365121 8034 raft_consensus.cc:1203] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.374792 11873 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.375031 11873 tablet_bootstrap.cc:492] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:04.375188 11873 ts_tablet_manager.cc:1367] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240503 08:00:04.375633 11873 raft_consensus.cc:348] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.375737 11873 raft_consensus.cc:729] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:04.375828 11873 consensus_queue.cc:260] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.376070 11873 ts_tablet_manager.cc:1398] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:04.376633 8054 tablet_copy_service.cc:342] P 61a9a0a13deb4d72b73f8623f6576766: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-2f40fae73f8f4c63b2d656ac880799e2 received from {username='slave'} at 127.0.99.137:39435
I20240503 08:00:04.376720 8054 tablet_copy_service.cc:433] P 61a9a0a13deb4d72b73f8623f6576766: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-2f40fae73f8f4c63b2d656ac880799e2 on tablet 2f40fae73f8f4c63b2d656ac880799e2 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:04.380143 11878 tablet_copy_client.cc:523] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.381317 11878 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 08:00:04.383456 11883 tablet_copy_client.cc:523] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.384603 11883 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 08:00:04.399518 11878 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.399791 11878 tablet_bootstrap.cc:492] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 08:00:04.399961 11878 ts_tablet_manager.cc:1367] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.001s
I20240503 08:00:04.400399 11878 raft_consensus.cc:348] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:04.400523 11878 raft_consensus.cc:729] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 08:00:04.400621 11878 consensus_queue.cc:260] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:04.400945 11878 ts_tablet_manager.cc:1398] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:00:04.401525 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-9e47575b08ad4f54b1f8d35db27d9e58 received from {username='slave'} at 127.0.99.134:54929
I20240503 08:00:04.401600 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-9e47575b08ad4f54b1f8d35db27d9e58 on tablet 9e47575b08ad4f54b1f8d35db27d9e58 with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 08:00:04.403694 11883 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.403918 11883 tablet_bootstrap.cc:492] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 08:00:04.404067 11883 ts_tablet_manager.cc:1367] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240503 08:00:04.404419 11883 raft_consensus.cc:348] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:04.404516 11883 raft_consensus.cc:729] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 08:00:04.404600 11883 consensus_queue.cc:260] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:04.404875 11883 ts_tablet_manager.cc:1398] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:00:04.405403 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-cd0045edfdb441f2ae6920b8a97555bb received from {username='slave'} at 127.0.99.135:56655
I20240503 08:00:04.405490 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-cd0045edfdb441f2ae6920b8a97555bb on tablet cd0045edfdb441f2ae6920b8a97555bb with peer cae9627a7600448faa8fcb7989b307ee
I20240503 08:00:04.405849 11888 tablet_copy_client.cc:523] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.407207 11888 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 08:00:04.408425 11893 tablet_copy_client.cc:523] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.409531 11893 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:04.421854 11898 tablet_copy_client.cc:523] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.422230 11901 tablet_copy_client.cc:523] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.423197 11898 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 08:00:04.423465 11901 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap starting.
I20240503 08:00:04.428207 11888 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.428440 11888 tablet_bootstrap.cc:492] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 08:00:04.428594 11888 ts_tablet_manager.cc:1367] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20240503 08:00:04.428982 11888 raft_consensus.cc:348] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:04.429193 11888 raft_consensus.cc:729] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 08:00:04.429297 11888 consensus_queue.cc:260] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:04.431021 11888 ts_tablet_manager.cc:1398] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:00:04.431437 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session 713b76caded84f269397165703739870-8efd24415e484e2a9af88b3aef6f78cc received from {username='slave'} at 127.0.99.136:47971
I20240503 08:00:04.431517 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session 713b76caded84f269397165703739870-8efd24415e484e2a9af88b3aef6f78cc on tablet 8efd24415e484e2a9af88b3aef6f78cc with peer 713b76caded84f269397165703739870
I20240503 08:00:04.435945 11908 tablet_copy_client.cc:523] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.437299 11908 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:04.438302 11893 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.438513 11893 tablet_bootstrap.cc:492] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:04.438649 11893 ts_tablet_manager.cc:1367] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.004s
I20240503 08:00:04.439013 11893 raft_consensus.cc:348] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.439106 11893 raft_consensus.cc:729] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:04.439195 11893 consensus_queue.cc:260] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.439440 11893 ts_tablet_manager.cc:1398] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:04.440009 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-885b10f956b54ebaa82614f8a1f979d1 received from {username='slave'} at 127.0.99.137:42151
I20240503 08:00:04.440083 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-885b10f956b54ebaa82614f8a1f979d1 on tablet 885b10f956b54ebaa82614f8a1f979d1 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:04.450517 11898 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.450788 11898 tablet_bootstrap.cc:492] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 08:00:04.450943 11898 ts_tablet_manager.cc:1367] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.000s
I20240503 08:00:04.451354 11898 raft_consensus.cc:348] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:04.451472 11898 raft_consensus.cc:729] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 08:00:04.451560 11898 consensus_queue.cc:260] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:04.452252 11898 ts_tablet_manager.cc:1398] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:00:04.453147 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 713b76caded84f269397165703739870-94f828890165466787015f21507e6ba2 received from {username='slave'} at 127.0.99.136:52377
I20240503 08:00:04.453238 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 713b76caded84f269397165703739870-94f828890165466787015f21507e6ba2 on tablet 94f828890165466787015f21507e6ba2 with peer 713b76caded84f269397165703739870
I20240503 08:00:04.463369 11901 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.463635 11901 tablet_bootstrap.cc:492] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: Bootstrap complete.
I20240503 08:00:04.463793 11901 ts_tablet_manager.cc:1367] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent bootstrapping tablet: real 0.040s user 0.020s sys 0.005s
I20240503 08:00:04.464203 11901 raft_consensus.cc:348] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:04.464331 11901 raft_consensus.cc:729] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Initialized, Role: LEARNER
I20240503 08:00:04.464422 11901 consensus_queue.cc:260] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: true } }
I20240503 08:00:04.464679 11901 ts_tablet_manager.cc:1398] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:04.465502 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-d74a03a296dc48e582faeddc7b978ccd received from {username='slave'} at 127.0.99.134:54879
I20240503 08:00:04.465585 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 66f9a693c3b3409dbcce8f3ee500bf5e-d74a03a296dc48e582faeddc7b978ccd on tablet d74a03a296dc48e582faeddc7b978ccd with peer 66f9a693c3b3409dbcce8f3ee500bf5e
I20240503 08:00:04.469777 11908 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.470031 11908 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:04.470185 11908 ts_tablet_manager.cc:1367] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.003s
I20240503 08:00:04.470558 11908 raft_consensus.cc:348] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.471089 11908 raft_consensus.cc:729] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:04.471185 11908 consensus_queue.cc:260] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.472108 11908 ts_tablet_manager.cc:1398] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:04.472666 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-08511a1b1eab4da9b810283d12cccee3 received from {username='slave'} at 127.0.99.137:38407
I20240503 08:00:04.472743 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-08511a1b1eab4da9b810283d12cccee3 on tablet 08511a1b1eab4da9b810283d12cccee3 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:04.487392 11913 tablet_copy_client.cc:523] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.488806 11913 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap starting.
I20240503 08:00:04.496492 11918 tablet_copy_client.cc:523] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.497872 11918 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: Bootstrap starting.
I20240503 08:00:04.514429 11913 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.514707 11913 tablet_bootstrap.cc:492] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: Bootstrap complete.
I20240503 08:00:04.514699 11923 tablet_copy_client.cc:523] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:04.514875 11913 ts_tablet_manager.cc:1367] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: Time spent bootstrapping tablet: real 0.026s user 0.025s sys 0.000s
I20240503 08:00:04.515331 11913 raft_consensus.cc:348] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:04.515458 11913 raft_consensus.cc:729] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61a9a0a13deb4d72b73f8623f6576766, State: Initialized, Role: LEARNER
I20240503 08:00:04.515550 11913 consensus_queue.cc:260] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: true } }
I20240503 08:00:04.515815 11913 ts_tablet_manager.cc:1398] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:04.516108 11923 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:04.516389 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session 61a9a0a13deb4d72b73f8623f6576766-64eb15b0505a4649a0043be00250c299 received from {username='slave'} at 127.0.99.133:54799
I20240503 08:00:04.516464 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session 61a9a0a13deb4d72b73f8623f6576766-64eb15b0505a4649a0043be00250c299 on tablet 64eb15b0505a4649a0043be00250c299 with peer 61a9a0a13deb4d72b73f8623f6576766
I20240503 08:00:04.527087 9336 raft_consensus.cc:1203] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:00:04.539223 11918 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.539536 11918 tablet_bootstrap.cc:492] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: Bootstrap complete.
I20240503 08:00:04.539714 11918 ts_tablet_manager.cc:1367] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.000s
I20240503 08:00:04.540165 11918 raft_consensus.cc:348] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:04.540302 11918 raft_consensus.cc:729] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 713b76caded84f269397165703739870, State: Initialized, Role: LEARNER
I20240503 08:00:04.540401 11918 consensus_queue.cc:260] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: true } }
I20240503 08:00:04.540726 11918 ts_tablet_manager.cc:1398] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:00:04.541478 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 713b76caded84f269397165703739870-d9e91e1b70504ac695eae61f72d3c149 received from {username='slave'} at 127.0.99.136:40651
I20240503 08:00:04.541556 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 713b76caded84f269397165703739870-d9e91e1b70504ac695eae61f72d3c149 on tablet d9e91e1b70504ac695eae61f72d3c149 with peer 713b76caded84f269397165703739870
I20240503 08:00:04.543047 11923 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:04.543294 11923 tablet_bootstrap.cc:492] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:04.543447 11923 ts_tablet_manager.cc:1367] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.005s
I20240503 08:00:04.543820 11923 raft_consensus.cc:348] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.543926 11923 raft_consensus.cc:729] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:04.544030 11923 consensus_queue.cc:260] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:04.544274 11923 ts_tablet_manager.cc:1398] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:04.544778 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-387f6b1606b94e11897372c0557870e4 received from {username='slave'} at 127.0.99.137:38407
I20240503 08:00:04.544845 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-387f6b1606b94e11897372c0557870e4 on tablet 387f6b1606b94e11897372c0557870e4 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:04.560616 8427 raft_consensus.cc:1203] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.618345 8427 raft_consensus.cc:1203] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.647737 8164 raft_consensus.cc:1203] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.674935 8427 raft_consensus.cc:1203] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.719338 8427 raft_consensus.cc:1203] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.725885 8295 raft_consensus.cc:1203] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.730830 9336 raft_consensus.cc:1203] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.747264 12421 raft_consensus.cc:1050] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 08:00:04.747712 12421 consensus_queue.cc:237] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:04.748979 8034 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.749361 6568 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.749406 8164 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.749788 12421 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.749974 12007 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.750254 12007 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.753051 12007 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:04.753456 8164 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:04.754053 8033 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:04.755216 6568 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:04.757354 6155 catalog_manager.cc:5484] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:04.796028 6437 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184"
dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54536
I20240503 08:00:04.796168 6437 raft_consensus.cc:595] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:04.811282 8164 raft_consensus.cc:1203] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.897266 8164 raft_consensus.cc:1203] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.927735 12503 raft_consensus.cc:1050] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:04.928223 12503 consensus_queue.cc:237] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:04.929500 6437 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:04.929515 6304 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:04.929576 9336 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1 [term 2 LEARNER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:04.929858 12503 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:04.930047 12505 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:04.930217 12238 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:04.932258 8033 raft_consensus.cc:1203] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.932188 12503 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.933236 6437 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.934127 6304 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.935727 9336 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.936076 6163 catalog_manager.cc:5484] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: config changed from index 8 to 9, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "61a9a0a13deb4d72b73f8623f6576766" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:04.940795 8033 consensus_queue.cc:237] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:04.941732 9336 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:04.941785 6304 raft_consensus.cc:1261] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Refusing update from remote peer 61a9a0a13deb4d72b73f8623f6576766: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:04.942085 12505 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:04.942279 12506 consensus_queue.cc:1035] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:04.943650 12238 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.944075 6304 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.944084 9336 raft_consensus.cc:2932] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.945130 6151 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f40fae73f8f4c63b2d656ac880799e2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:04.946180 6155 catalog_manager.cc:5484] T 2f40fae73f8f4c63b2d656ac880799e2 P 0394d429c78744aca1220af8f39b11c1 reported cstate change: config changed from index 9 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "61a9a0a13deb4d72b73f8623f6576766" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.948014 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f40fae73f8f4c63b2d656ac880799e2 with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34676
I20240503 08:00:04.948453 12510 tablet_replica.cc:330] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 08:00:04.948577 12510 raft_consensus.cc:2229] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:04.948765 12510 raft_consensus.cc:2258] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:04.949287 12510 ts_tablet_manager.cc:1875] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:04.950827 12510 ts_tablet_manager.cc:1888] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:00:04.950906 12510 log.cc:1196] T 2f40fae73f8f4c63b2d656ac880799e2 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2f40fae73f8f4c63b2d656ac880799e2
I20240503 08:00:04.951329 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet 2f40fae73f8f4c63b2d656ac880799e2 (table rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]) successfully deleted
I20240503 08:00:04.960381 12387 raft_consensus.cc:1050] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:04.960783 12387 consensus_queue.cc:237] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:04.960942 9336 raft_consensus.cc:1203] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:04.962132 9336 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.962154 6437 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.962169 8164 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.962541 12511 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.962728 12387 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.963234 12161 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.964841 12387 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.965277 8164 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.965305 9336 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.967103 6164 catalog_manager.cc:5484] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:04.968612 6437 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:04.991827 12398 raft_consensus.cc:1050] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 08:00:04.992290 12398 consensus_queue.cc:237] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:04.993726 6437 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.993876 8033 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.993891 8427 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.994215 12517 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.994423 12004 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.994580 12519 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.996237 12430 raft_consensus.cc:1050] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 08:00:04.996626 12430 consensus_queue.cc:237] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:04.998093 8033 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.998088 8428 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.998090 6568 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:04.998427 12008 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.998623 12431 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.998826 12524 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:04.999068 12004 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.000619 12431 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.001300 8428 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.001390 8033 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.001767 6163 catalog_manager.cc:5484] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.002481 6437 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.003669 8033 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.005079 6164 catalog_manager.cc:5484] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.005456 6568 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.006196 6698 consensus_queue.cc:237] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:05.008747 8428 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.009227 12519 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.009522 8428 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.009742 8033 raft_consensus.cc:1261] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.010923 12517 consensus_queue.cc:1035] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.011361 12398 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.011705 8427 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.011901 6304 consensus_queue.cc:237] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:05.012864 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8efd24415e484e2a9af88b3aef6f78cc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.014057 6157 catalog_manager.cc:5484] T 8efd24415e484e2a9af88b3aef6f78cc P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.015507 8427 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.015867 12524 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.017004 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet 8efd24415e484e2a9af88b3aef6f78cc with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34676
I20240503 08:00:05.017235 12510 tablet_replica.cc:330] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 08:00:05.017375 12510 raft_consensus.cc:2229] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.017560 12510 raft_consensus.cc:2258] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.017463 12430 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.018170 6568 raft_consensus.cc:1261] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.018397 12510 ts_tablet_manager.cc:1875] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.018540 12524 consensus_queue.cc:1035] T 94f828890165466787015f21507e6ba2 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.019721 8427 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.019789 8033 raft_consensus.cc:2932] T 8efd24415e484e2a9af88b3aef6f78cc P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.020460 12510 ts_tablet_manager.cc:1888] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:05.020519 12510 log.cc:1196] T 8efd24415e484e2a9af88b3aef6f78cc P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8efd24415e484e2a9af88b3aef6f78cc
I20240503 08:00:05.020548 6568 raft_consensus.cc:2932] T 94f828890165466787015f21507e6ba2 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.020934 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 94f828890165466787015f21507e6ba2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.022699 6157 catalog_manager.cc:5484] T 94f828890165466787015f21507e6ba2 P 713b76caded84f269397165703739870 reported cstate change: config changed from index 8 to 9, VOTER 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.022716 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet 8efd24415e484e2a9af88b3aef6f78cc (table rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]) successfully deleted
I20240503 08:00:05.025126 8014 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f828890165466787015f21507e6ba2 with delete_type TABLET_DATA_TOMBSTONED (TS 61a9a0a13deb4d72b73f8623f6576766 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59552
I20240503 08:00:05.025578 12528 tablet_replica.cc:330] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: stopping tablet replica
I20240503 08:00:05.025851 12528 raft_consensus.cc:2229] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.026072 12528 raft_consensus.cc:2258] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.026880 12528 ts_tablet_manager.cc:1875] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.028640 12528 ts_tablet_manager.cc:1888] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:05.028721 12528 log.cc:1196] T 94f828890165466787015f21507e6ba2 P 61a9a0a13deb4d72b73f8623f6576766: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/94f828890165466787015f21507e6ba2
I20240503 08:00:05.029202 6151 catalog_manager.cc:4830] TS 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285): tablet 94f828890165466787015f21507e6ba2 (table rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]) successfully deleted
I20240503 08:00:05.047088 9336 raft_consensus.cc:1203] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:05.076349 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58838
I20240503 08:00:05.076467 6568 raft_consensus.cc:595] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.085971 6437 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184"
dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54588
I20240503 08:00:05.086109 6437 raft_consensus.cc:595] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.122839 6304 consensus_queue.cc:237] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:05.124258 8427 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:05.124249 6568 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:05.124612 12008 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:00:05.124841 6240 consensus_peers.cc:485] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e03d7158b64f434a8871f009864a0b72. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:05.124842 12524 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:05.126616 12430 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.127017 8427 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.128356 6568 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.129801 6164 catalog_manager.cc:5484] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 reported cstate change: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.165470 12398 raft_consensus.cc:1050] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 08:00:05.165835 12398 consensus_queue.cc:237] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:05.167261 8033 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.167280 6304 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.167551 8427 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.167686 12004 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.168002 12517 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.168346 12004 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.169394 12513 raft_consensus.cc:1050] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 08:00:05.169785 12513 consensus_queue.cc:237] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:05.171304 6437 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.171332 8295 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.171720 12387 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.172005 12387 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.173624 8164 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.173956 12517 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.174336 8427 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.176198 12513 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.176127 6304 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.176863 6156 catalog_manager.cc:5484] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870 reported cstate change: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.181344 12513 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.181340 8033 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.182554 6437 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.184300 12511 raft_consensus.cc:1050] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:05.184358 6698 consensus_queue.cc:237] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:05.184648 12511 consensus_queue.cc:237] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:05.186098 12420 raft_consensus.cc:979] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: : Instructing follower 66f9a693c3b3409dbcce8f3ee500bf5e to start an election
I20240503 08:00:05.186136 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58874
I20240503 08:00:05.186199 12420 raft_consensus.cc:1067] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Signalling peer 66f9a693c3b3409dbcce8f3ee500bf5e to start an election
I20240503 08:00:05.186232 6568 raft_consensus.cc:595] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.186645 8165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184"
dest_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e"
from {username='slave'} at 127.0.99.130:41805
I20240503 08:00:05.186793 6304 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.186800 8165 raft_consensus.cc:482] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:05.186892 8165 raft_consensus.cc:3037] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.187271 9336 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.187471 12671 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.187707 12161 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.187805 8165 raft_consensus.cc:504] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:05.187981 8295 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.188931 8165 leader_election.cc:288] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [CANDIDATE]: Term 2 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133:41285)
I20240503 08:00:05.189334 8034 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.189633 8164 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.189806 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184" candidate_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 08:00:05.189905 6437 raft_consensus.cc:3032] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:05.189953 6437 raft_consensus.cc:729] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 08:00:05.190090 6437 consensus_queue.cc:260] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:05.190268 6437 raft_consensus.cc:3037] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.190984 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184" candidate_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 08:00:05.191083 6568 raft_consensus.cc:3037] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.191164 6437 raft_consensus.cc:2454] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f9a693c3b3409dbcce8f3ee500bf5e in term 2.
I20240503 08:00:05.191452 12398 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.191951 6568 raft_consensus.cc:2454] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f9a693c3b3409dbcce8f3ee500bf5e in term 2.
I20240503 08:00:05.192510 8427 raft_consensus.cc:1261] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.192761 8100 leader_election.cc:302] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5, 66f9a693c3b3409dbcce8f3ee500bf5e; no voters:
I20240503 08:00:05.193008 12398 consensus_queue.cc:1035] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.193253 12317 raft_consensus.cc:2781] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:05.193403 12317 raft_consensus.cc:686] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 LEADER]: Becoming Leader. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Running, Role: LEADER
I20240503 08:00:05.193429 12004 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.193686 12317 consensus_queue.cc:237] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:05.195210 6164 catalog_manager.cc:5484] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.197328 8034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b7dcf55364209bd26129488dd0184" candidate_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "61a9a0a13deb4d72b73f8623f6576766"
I20240503 08:00:05.197433 8034 raft_consensus.cc:3037] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.197499 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d7012d8274214cf58c70d52062474b3a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.197611 6163 catalog_manager.cc:5484] T d7012d8274214cf58c70d52062474b3a P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.197855 8032 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.197958 8427 raft_consensus.cc:2932] T d7012d8274214cf58c70d52062474b3a P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.198356 8034 raft_consensus.cc:2454] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f9a693c3b3409dbcce8f3ee500bf5e in term 2.
I20240503 08:00:05.200568 8033 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.200989 12513 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.201807 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet d7012d8274214cf58c70d52062474b3a with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35598
I20240503 08:00:05.201838 6155 catalog_manager.cc:5484] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: term changed from 1 to 2, leader changed from 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) to 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134). New cstate: current_term: 2 leader_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.202709 12682 tablet_replica.cc:330] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 08:00:05.202900 12682 raft_consensus.cc:2229] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.202775 12513 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.203248 12682 raft_consensus.cc:2258] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.204423 8033 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.204703 9336 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.207145 6157 catalog_manager.cc:5484] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 reported cstate change: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.207332 6304 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.207718 12682 ts_tablet_manager.cc:1875] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.210093 12682 ts_tablet_manager.cc:1888] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:05.210235 12682 log.cc:1196] T d7012d8274214cf58c70d52062474b3a P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d7012d8274214cf58c70d52062474b3a
I20240503 08:00:05.210860 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet d7012d8274214cf58c70d52062474b3a (table rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]) successfully deleted
I20240503 08:00:05.213585 6568 consensus_queue.cc:237] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:05.214927 8033 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.215371 9336 raft_consensus.cc:1261] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.215785 12511 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.215909 12387 consensus_queue.cc:1035] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.217693 12671 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.219082 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 885b10f956b54ebaa82614f8a1f979d1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.219558 8033 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.220027 9336 raft_consensus.cc:2932] T 885b10f956b54ebaa82614f8a1f979d1 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.221695 6163 catalog_manager.cc:5484] T 885b10f956b54ebaa82614f8a1f979d1 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.223780 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 885b10f956b54ebaa82614f8a1f979d1 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35598
I20240503 08:00:05.224004 12682 tablet_replica.cc:330] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 08:00:05.224134 12682 raft_consensus.cc:2229] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.224298 12682 raft_consensus.cc:2258] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.224928 12682 ts_tablet_manager.cc:1875] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.226668 12682 ts_tablet_manager.cc:1888] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:05.226742 12682 log.cc:1196] T 885b10f956b54ebaa82614f8a1f979d1 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/885b10f956b54ebaa82614f8a1f979d1
I20240503 08:00:05.227296 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 885b10f956b54ebaa82614f8a1f979d1 (table rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]) successfully deleted
I20240503 08:00:05.246634 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67b9602786f8422d962885dda25c6c85"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58882
I20240503 08:00:05.246745 6568 raft_consensus.cc:595] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.259680 12723 raft_consensus.cc:1050] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 713b76caded84f269397165703739870 to VOTER
I20240503 08:00:05.260113 12723 consensus_queue.cc:237] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:05.261660 8165 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.261741 6304 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.261940 8427 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.262496 12421 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.262646 12420 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.263289 12421 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.265269 12723 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.265694 8165 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.266480 8427 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.267648 6304 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.268091 6157 catalog_manager.cc:5484] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 7 to 8, 713b76caded84f269397165703739870 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:05.297333 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58918
I20240503 08:00:05.297456 6568 raft_consensus.cc:595] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.305039 12732 raft_consensus.cc:1050] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 08:00:05.305502 12732 consensus_queue.cc:237] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:05.307015 6304 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.307014 8165 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.307339 8033 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.307713 12421 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.307863 12732 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.307965 12420 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.309695 12511 raft_consensus.cc:1050] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 08:00:05.310091 12511 consensus_queue.cc:237] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:05.311595 12723 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.311868 8165 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.312038 6698 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.312232 12671 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.312431 12161 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.312750 8295 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.313432 6304 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.313948 8164 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.314250 8033 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.315071 12387 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.316035 6156 catalog_manager.cc:5484] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.320461 6437 consensus_queue.cc:237] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:05.320861 12161 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:05.321457 12431 raft_consensus.cc:1050] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 66f9a693c3b3409dbcce8f3ee500bf5e to VOTER
I20240503 08:00:05.321426 8165 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:05.321784 12431 consensus_queue.cc:237] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:05.323508 8165 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.323637 8033 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.323743 6155 catalog_manager.cc:5484] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 7 to 8, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.324039 12420 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.324416 12430 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.324425 8296 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:05.325269 6437 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.325620 12430 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.326256 8164 raft_consensus.cc:1261] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.326628 12420 consensus_queue.cc:1035] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.327934 12420 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.328523 8164 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.329594 8295 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.329988 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d74a03a296dc48e582faeddc7b978ccd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.330520 6163 catalog_manager.cc:5484] T d74a03a296dc48e582faeddc7b978ccd P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.330852 6568 consensus_queue.cc:237] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:05.331003 12430 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.332597 8295 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.333259 12387 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.334287 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet d74a03a296dc48e582faeddc7b978ccd with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35598
I20240503 08:00:05.334463 8164 raft_consensus.cc:1261] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.334861 12682 tablet_replica.cc:330] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 08:00:05.335021 12682 raft_consensus.cc:2229] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.335000 12387 consensus_queue.cc:1035] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.335173 12682 raft_consensus.cc:2258] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.336020 12682 ts_tablet_manager.cc:1875] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.336565 12511 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:05.337103 8164 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:05.338156 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd0045edfdb441f2ae6920b8a97555bb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.338167 12682 ts_tablet_manager.cc:1888] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:05.338234 12682 log.cc:1196] T d74a03a296dc48e582faeddc7b978ccd P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d74a03a296dc48e582faeddc7b978ccd
I20240503 08:00:05.338459 8033 raft_consensus.cc:2932] T d74a03a296dc48e582faeddc7b978ccd P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.338688 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet d74a03a296dc48e582faeddc7b978ccd (table rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]) successfully deleted
I20240503 08:00:05.340945 6155 catalog_manager.cc:5484] T cd0045edfdb441f2ae6920b8a97555bb P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 8 to 9, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.344157 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet cd0045edfdb441f2ae6920b8a97555bb with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45994
I20240503 08:00:05.345266 12787 tablet_replica.cc:330] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 08:00:05.346309 12430 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.346864 8165 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.348480 6437 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.349347 6157 catalog_manager.cc:5484] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.349505 8296 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.352041 8295 raft_consensus.cc:2932] T cd0045edfdb441f2ae6920b8a97555bb P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:05.354080 6304 consensus_queue.cc:237] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:05.355296 8165 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.355665 12431 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.356664 8295 raft_consensus.cc:1261] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.357033 12431 consensus_queue.cc:1035] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.359234 12431 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.360663 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e47575b08ad4f54b1f8d35db27d9e58 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.360725 8165 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.361086 8295 raft_consensus.cc:2932] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.363137 6163 catalog_manager.cc:5484] T 9e47575b08ad4f54b1f8d35db27d9e58 P cae9627a7600448faa8fcb7989b307ee reported cstate change: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:05.364907 12787 raft_consensus.cc:2229] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.365157 12787 pending_rounds.cc:62] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: Trying to abort 1 pending ops.
I20240503 08:00:05.365271 12787 pending_rounds.cc:68] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023506248949006336 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "cd0045edfdb441f2ae6920b8a97555bb" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } } }
I20240503 08:00:05.365509 12787 raft_consensus.cc:2894] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 08:00:05.365612 12787 raft_consensus.cc:2866] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:05.365697 12787 raft_consensus.cc:2258] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.366277 12787 ts_tablet_manager.cc:1875] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.366603 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e47575b08ad4f54b1f8d35db27d9e58 with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34676
I20240503 08:00:05.367058 12510 tablet_replica.cc:330] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 08:00:05.367228 12510 raft_consensus.cc:2229] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.367421 12510 raft_consensus.cc:2258] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.368141 12510 ts_tablet_manager.cc:1875] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.369390 12787 ts_tablet_manager.cc:1888] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:05.369460 12787 log.cc:1196] T cd0045edfdb441f2ae6920b8a97555bb P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/cd0045edfdb441f2ae6920b8a97555bb
I20240503 08:00:05.369889 12510 ts_tablet_manager.cc:1888] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:05.370226 12510 log.cc:1196] T 9e47575b08ad4f54b1f8d35db27d9e58 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9e47575b08ad4f54b1f8d35db27d9e58
I20240503 08:00:05.370779 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet 9e47575b08ad4f54b1f8d35db27d9e58 (table rebalance_test_table_3 [id=a3e379ee49974a789420fd03e3f1edce]) successfully deleted
I20240503 08:00:05.370878 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet cd0045edfdb441f2ae6920b8a97555bb (table rebalance_test_table_0 [id=599c32d0acb7434e8b25b6c8a2445def]) successfully deleted
I20240503 08:00:05.380075 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67b9602786f8422d962885dda25c6c85"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58934
I20240503 08:00:05.380180 6568 raft_consensus.cc:595] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.386834 6568 consensus_queue.cc:237] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:05.388010 8033 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:05.388064 6304 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:05.388125 6502 consensus_peers.cc:485] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d608390b027432f8a3f89130c051759. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:05.388459 12513 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:05.388633 12387 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:05.390223 12511 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.390622 8033 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.391530 6304 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.392431 6155 catalog_manager.cc:5484] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.407846 6437 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149"
dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54666
I20240503 08:00:05.407960 6437 raft_consensus.cc:595] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.435233 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58974
I20240503 08:00:05.435354 6568 raft_consensus.cc:595] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.448971 12387 raft_consensus.cc:979] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: : Instructing follower 66f9a693c3b3409dbcce8f3ee500bf5e to start an election
I20240503 08:00:05.449206 12387 raft_consensus.cc:1067] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Signalling peer 66f9a693c3b3409dbcce8f3ee500bf5e to start an election
I20240503 08:00:05.449865 8165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4"
dest_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e"
from {username='slave'} at 127.0.99.131:58645
I20240503 08:00:05.449992 8165 raft_consensus.cc:482] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:05.450053 8165 raft_consensus.cc:3037] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.450995 8165 raft_consensus.cc:504] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:05.451509 8165 leader_election.cc:288] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [CANDIDATE]: Term 2 election: Requested vote from peers 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867)
I20240503 08:00:05.452050 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4" candidate_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 08:00:05.452179 6437 raft_consensus.cc:3037] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.452942 6437 raft_consensus.cc:2454] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f9a693c3b3409dbcce8f3ee500bf5e in term 2.
I20240503 08:00:05.453583 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4" candidate_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 08:00:05.453697 6568 raft_consensus.cc:3032] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:05.453732 6568 raft_consensus.cc:729] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 08:00:05.453859 6568 consensus_queue.cc:260] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:05.454288 6568 raft_consensus.cc:3037] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.455225 6568 raft_consensus.cc:2454] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f9a693c3b3409dbcce8f3ee500bf5e in term 2.
I20240503 08:00:05.455572 8100 leader_election.cc:302] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5, 66f9a693c3b3409dbcce8f3ee500bf5e; no voters:
I20240503 08:00:05.455752 12679 raft_consensus.cc:2781] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:05.455828 12679 raft_consensus.cc:686] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 LEADER]: Becoming Leader. State: Replica: 66f9a693c3b3409dbcce8f3ee500bf5e, State: Running, Role: LEADER
I20240503 08:00:05.455955 12679 consensus_queue.cc:237] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:05.457973 9336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8625e8b653624ea6acf6e04c3fd352a4" candidate_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0394d429c78744aca1220af8f39b11c1"
I20240503 08:00:05.458163 9336 raft_consensus.cc:3037] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.458042 6163 catalog_manager.cc:5484] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: term changed from 1 to 2, leader changed from 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) to 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134). New cstate: current_term: 2 leader_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.459185 9336 raft_consensus.cc:2454] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f9a693c3b3409dbcce8f3ee500bf5e in term 2.
I20240503 08:00:05.473433 6568 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67b9602786f8422d962885dda25c6c85"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58994
I20240503 08:00:05.473558 6568 raft_consensus.cc:595] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.496479 12524 raft_consensus.cc:1050] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:05.496573 6437 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149"
dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54686
I20240503 08:00:05.496716 6437 raft_consensus.cc:595] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.496913 12524 consensus_queue.cc:237] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:05.501582 6698 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.501968 12431 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.502801 8295 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.503203 12430 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.503365 9336 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.503826 12430 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.505020 12008 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.505403 8295 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.506842 6698 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.507421 6163 catalog_manager.cc:5484] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.512104 12519 raft_consensus.cc:1050] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER 61a9a0a13deb4d72b73f8623f6576766 to VOTER
I20240503 08:00:05.512516 12519 consensus_queue.cc:237] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:05.513916 8427 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.514225 6566 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.514593 8033 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.514787 12517 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.514971 12004 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.515270 12398 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.517721 9336 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.518055 12519 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:05.519770 8033 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:05.519919 6304 consensus_queue.cc:237] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:05.520442 6157 catalog_manager.cc:5484] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.521234 6566 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:05.522400 8295 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.523160 12524 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.523806 8427 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 61a9a0a13deb4d72b73f8623f6576766 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:05.524644 12430 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.526628 8296 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.527143 6164 catalog_manager.cc:5484] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 reported cstate change: config changed from index 8 to 9, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.527952 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 387f6b1606b94e11897372c0557870e4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.530380 9336 raft_consensus.cc:1261] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.531087 12524 consensus_queue.cc:1035] T 387f6b1606b94e11897372c0557870e4 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.531947 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet 387f6b1606b94e11897372c0557870e4 with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45994
I20240503 08:00:05.532159 12787 tablet_replica.cc:330] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 08:00:05.532294 12787 raft_consensus.cc:2229] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.532480 12787 raft_consensus.cc:2258] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.532446 6698 consensus_queue.cc:237] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:05.533188 12787 ts_tablet_manager.cc:1875] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.533433 9336 raft_consensus.cc:2932] T 387f6b1606b94e11897372c0557870e4 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.534392 8427 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.534818 12398 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.535360 12787 ts_tablet_manager.cc:1888] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:05.535441 12787 log.cc:1196] T 387f6b1606b94e11897372c0557870e4 P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/387f6b1606b94e11897372c0557870e4
I20240503 08:00:05.535976 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet 387f6b1606b94e11897372c0557870e4 (table rebalance_test_table_8 [id=1a0ab13880ed40718b66ed0bd3121c49]) successfully deleted
I20240503 08:00:05.536047 8033 raft_consensus.cc:1261] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:05.536473 12519 consensus_queue.cc:1035] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.536959 12398 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:05.537014 8427 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:05.537459 8033 raft_consensus.cc:2932] T 64eb15b0505a4649a0043be00250c299 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:05.538290 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64eb15b0505a4649a0043be00250c299 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:05.539438 6163 catalog_manager.cc:5484] T 64eb15b0505a4649a0043be00250c299 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 8 to 9, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.542318 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet 64eb15b0505a4649a0043be00250c299 with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41936
I20240503 08:00:05.543170 12920 tablet_replica.cc:330] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 08:00:05.543604 12920 raft_consensus.cc:2229] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:05.543812 12920 raft_consensus.cc:2258] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:05.544413 12920 ts_tablet_manager.cc:1875] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:05.546617 12920 ts_tablet_manager.cc:1888] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:05.546725 12920 log.cc:1196] T 64eb15b0505a4649a0043be00250c299 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/64eb15b0505a4649a0043be00250c299
I20240503 08:00:05.547204 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet 64eb15b0505a4649a0043be00250c299 (table rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]) successfully deleted
I20240503 08:00:05.566569 6566 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67b9602786f8422d962885dda25c6c85"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59020
I20240503 08:00:05.566701 6566 raft_consensus.cc:595] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.571440 6437 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:05.571888 12679 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.574249 6437 consensus_queue.cc:237] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:05.575822 6304 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:05.575855 8296 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:05.576246 12723 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:05.576530 12723 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:05.578403 12421 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.579384 8296 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.580081 6304 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:05.581485 6157 catalog_manager.cc:5484] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:05.582711 6371 consensus_peers.cc:485] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4925e30c2fb74934bbd5eb7415193c96. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:05.583550 12524 raft_consensus.cc:1050] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:05.583930 12524 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:05.585590 9336 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.585693 8427 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.586110 12961 ts_tablet_manager.cc:902] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer 253dd189c3c542248402cba222394f43 (127.0.99.129:38721)
I20240503 08:00:05.586091 12430 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.586429 6698 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:05.586663 12961 tablet_copy_client.cc:308] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:38721
I20240503 08:00:05.587401 12008 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.587697 12008 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:05.588306 6326 tablet_copy_service.cc:140] P 253dd189c3c542248402cba222394f43: Received BeginTabletCopySession request for tablet e03d7158b64f434a8871f009864a0b72 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:38407)
I20240503 08:00:05.588387 6326 tablet_copy_service.cc:161] P 253dd189c3c542248402cba222394f43: Beginning new tablet copy session on tablet e03d7158b64f434a8871f009864a0b72 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:38407: session id = 0394d429c78744aca1220af8f39b11c1-e03d7158b64f434a8871f009864a0b72
I20240503 08:00:05.589253 6326 tablet_copy_source_session.cc:205] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:05.589638 12430 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.590268 6698 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.590489 12961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03d7158b64f434a8871f009864a0b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.591385 8427 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.592377 12961 tablet_copy_client.cc:783] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:05.592543 12961 tablet_copy_client.cc:647] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:05.593014 6163 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.594661 12962 tablet_copy_client.cc:1002] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:05.594610 9335 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:05.607719 6437 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149"
dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54716
I20240503 08:00:05.607827 6437 raft_consensus.cc:595] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.635000 6304 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3"
dest_uuid: "253dd189c3c542248402cba222394f43"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49794
I20240503 08:00:05.635149 6304 raft_consensus.cc:595] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.644062 6566 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67b9602786f8422d962885dda25c6c85"
dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59060
I20240503 08:00:05.644196 6566 raft_consensus.cc:595] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.662370 12387 raft_consensus.cc:979] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: : Instructing follower 02fe52b383f3498ebbbe4542c8cc12fb to start an election
I20240503 08:00:05.662477 12387 raft_consensus.cc:1067] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Signalling peer 02fe52b383f3498ebbbe4542c8cc12fb to start an election
I20240503 08:00:05.662873 6436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149"
dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54752
I20240503 08:00:05.662884 6437 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67b9602786f8422d962885dda25c6c85"
dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
from {username='slave'} at 127.0.99.131:43529
I20240503 08:00:05.662992 6436 raft_consensus.cc:595] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.663028 6437 raft_consensus.cc:482] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:05.663098 6437 raft_consensus.cc:3037] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.664084 6437 raft_consensus.cc:504] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:05.664525 6437 leader_election.cc:288] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 2 election: Requested vote from peers 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099), cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381), 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851)
I20240503 08:00:05.664793 6566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9602786f8422d962885dda25c6c85" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "34abd76ae3764e7f9641eaddaffdd5a5"
I20240503 08:00:05.664922 6566 raft_consensus.cc:3032] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:05.664958 6566 raft_consensus.cc:729] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 34abd76ae3764e7f9641eaddaffdd5a5, State: Running, Role: LEADER
I20240503 08:00:05.665139 6566 consensus_queue.cc:260] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:05.665251 8296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9602786f8422d962885dda25c6c85" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "cae9627a7600448faa8fcb7989b307ee"
I20240503 08:00:05.665316 6566 raft_consensus.cc:3037] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.665424 8296 raft_consensus.cc:3037] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.666205 6566 raft_consensus.cc:2454] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 2.
I20240503 08:00:05.666353 8296 raft_consensus.cc:2454] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 2.
I20240503 08:00:05.666661 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b9602786f8422d962885dda25c6c85" candidate_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e"
I20240503 08:00:05.666792 8165 raft_consensus.cc:3037] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.667666 8165 raft_consensus.cc:2454] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fe52b383f3498ebbbe4542c8cc12fb in term 2.
I20240503 08:00:05.667970 6372 leader_election.cc:302] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 34abd76ae3764e7f9641eaddaffdd5a5, cae9627a7600448faa8fcb7989b307ee; no voters:
I20240503 08:00:05.668123 12421 raft_consensus.cc:2781] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:05.668231 12421 raft_consensus.cc:686] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 LEADER]: Becoming Leader. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 08:00:05.668357 12421 consensus_queue.cc:237] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:05.670158 6157 catalog_manager.cc:5484] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: term changed from 1 to 2, leader changed from 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) to 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130). New cstate: current_term: 2 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:05.693048 6304 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3"
dest_uuid: "253dd189c3c542248402cba222394f43"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49824
I20240503 08:00:05.693230 6304 raft_consensus.cc:595] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.709295 12421 raft_consensus.cc:979] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: : Instructing follower 713b76caded84f269397165703739870 to start an election
I20240503 08:00:05.709378 12421 raft_consensus.cc:1067] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Signalling peer 713b76caded84f269397165703739870 to start an election
I20240503 08:00:05.709760 8427 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149"
dest_uuid: "713b76caded84f269397165703739870"
from {username='slave'} at 127.0.99.130:57767
I20240503 08:00:05.709908 8427 raft_consensus.cc:482] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:05.709983 8427 raft_consensus.cc:3037] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.710693 8427 raft_consensus.cc:504] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:05.711709 8427 leader_election.cc:288] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [CANDIDATE]: Term 2 election: Requested vote from peers 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351), 66f9a693c3b3409dbcce8f3ee500bf5e (127.0.99.134:42851)
I20240503 08:00:05.711907 6304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149" candidate_uuid: "713b76caded84f269397165703739870" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 08:00:05.712023 6304 raft_consensus.cc:3037] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.712168 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149" candidate_uuid: "713b76caded84f269397165703739870" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "02fe52b383f3498ebbbe4542c8cc12fb"
I20240503 08:00:05.712291 6437 raft_consensus.cc:3032] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:05.712330 6437 raft_consensus.cc:729] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 02fe52b383f3498ebbbe4542c8cc12fb, State: Running, Role: LEADER
I20240503 08:00:05.712450 6437 consensus_queue.cc:260] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:05.712608 6437 raft_consensus.cc:3037] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.712967 6304 raft_consensus.cc:2454] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713b76caded84f269397165703739870 in term 2.
I20240503 08:00:05.713505 6437 raft_consensus.cc:2454] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713b76caded84f269397165703739870 in term 2.
I20240503 08:00:05.713742 8361 leader_election.cc:302] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02fe52b383f3498ebbbe4542c8cc12fb, 253dd189c3c542248402cba222394f43, 713b76caded84f269397165703739870; no voters:
I20240503 08:00:05.713951 12300 raft_consensus.cc:2781] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:05.714125 12300 raft_consensus.cc:686] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 2 LEADER]: Becoming Leader. State: Replica: 713b76caded84f269397165703739870, State: Running, Role: LEADER
I20240503 08:00:05.714413 12300 consensus_queue.cc:237] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:05.715368 8165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e91e1b70504ac695eae61f72d3c149" candidate_uuid: "713b76caded84f269397165703739870" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e"
I20240503 08:00:05.715484 8165 raft_consensus.cc:3037] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:05.716214 8165 raft_consensus.cc:2454] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713b76caded84f269397165703739870 in term 2.
I20240503 08:00:05.716776 6157 catalog_manager.cc:5484] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 reported cstate change: term changed from 1 to 2, leader changed from 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) to 713b76caded84f269397165703739870 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "713b76caded84f269397165703739870" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:05.724997 8033 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:05.725396 12679 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.727777 6566 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:05.728237 12679 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.937600 6304 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3"
dest_uuid: "253dd189c3c542248402cba222394f43"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44260
I20240503 08:00:05.937718 6304 raft_consensus.cc:595] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:05.971019 9335 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:05.971424 12679 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:05.986055 13128 ts_tablet_manager.cc:902] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 08:00:05.986496 13128 tablet_copy_client.cc:308] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 08:00:05.986894 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet 7d608390b027432f8a3f89130c051759 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:42151)
I20240503 08:00:05.986984 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet 7d608390b027432f8a3f89130c051759 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:42151: session id = 0394d429c78744aca1220af8f39b11c1-7d608390b027432f8a3f89130c051759
I20240503 08:00:05.987603 6588 tablet_copy_source_session.cc:205] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:05.988329 13128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d608390b027432f8a3f89130c051759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:05.989894 13128 tablet_copy_client.cc:783] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:05.990022 13128 tablet_copy_client.cc:647] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:05.990880 13132 tablet_copy_client.cc:1002] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:06.037962 6437 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.038331 12679 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.040611 6568 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.041132 12317 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.050467 8165 consensus_queue.cc:237] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:06.051354 6437 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:06.051383 9335 raft_consensus.cc:1261] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:06.051705 13135 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:06.051833 13077 consensus_queue.cc:1035] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:06.053274 12317 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:06.053661 6437 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:06.055225 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8625e8b653624ea6acf6e04c3fd352a4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:06.056264 6157 catalog_manager.cc:5484] T 8625e8b653624ea6acf6e04c3fd352a4 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 8 to 10, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:06.056875 9335 raft_consensus.cc:2932] T 8625e8b653624ea6acf6e04c3fd352a4 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:06.058557 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet 8625e8b653624ea6acf6e04c3fd352a4 with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41936
I20240503 08:00:06.059794 13138 tablet_replica.cc:330] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 08:00:06.059996 13138 raft_consensus.cc:2229] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:06.060118 13138 raft_consensus.cc:2258] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:06.060701 13138 ts_tablet_manager.cc:1875] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:06.062359 13138 ts_tablet_manager.cc:1888] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:00:06.062420 13138 log.cc:1196] T 8625e8b653624ea6acf6e04c3fd352a4 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8625e8b653624ea6acf6e04c3fd352a4
I20240503 08:00:06.062834 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet 8625e8b653624ea6acf6e04c3fd352a4 (table rebalance_test_table_7 [id=d725f47438d04b388125b18a6d446148]) successfully deleted
I20240503 08:00:06.066174 12008 raft_consensus.cc:979] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: : Instructing follower 713b76caded84f269397165703739870 to start an election
I20240503 08:00:06.066257 12008 raft_consensus.cc:1067] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Signalling peer 713b76caded84f269397165703739870 to start an election
I20240503 08:00:06.066722 8427 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3"
dest_uuid: "713b76caded84f269397165703739870"
from {username='slave'} at 127.0.99.129:58653
I20240503 08:00:06.066831 8427 raft_consensus.cc:482] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:06.066887 8427 raft_consensus.cc:3037] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:06.067543 8427 raft_consensus.cc:504] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:06.068011 8427 leader_election.cc:288] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [CANDIDATE]: Term 2 election: Requested vote from peers e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367), 253dd189c3c542248402cba222394f43 (127.0.99.129:38721), 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867)
I20240503 08:00:06.068295 6698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3" candidate_uuid: "713b76caded84f269397165703739870" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e9976bc168264eaeba1d180fb38b5113"
I20240503 08:00:06.068362 6304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3" candidate_uuid: "713b76caded84f269397165703739870" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "253dd189c3c542248402cba222394f43"
I20240503 08:00:06.068421 6698 raft_consensus.cc:3037] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:06.068468 6304 raft_consensus.cc:3032] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:06.068506 6304 raft_consensus.cc:729] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 253dd189c3c542248402cba222394f43, State: Running, Role: LEADER
I20240503 08:00:06.068642 6304 consensus_queue.cc:260] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:06.068825 6304 raft_consensus.cc:3037] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:06.069404 6698 raft_consensus.cc:2454] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713b76caded84f269397165703739870 in term 2.
I20240503 08:00:06.069588 6304 raft_consensus.cc:2454] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713b76caded84f269397165703739870 in term 2.
I20240503 08:00:06.069903 8362 leader_election.cc:302] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 253dd189c3c542248402cba222394f43, 713b76caded84f269397165703739870, e9976bc168264eaeba1d180fb38b5113; no voters:
I20240503 08:00:06.070335 12300 raft_consensus.cc:2781] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:06.070422 12300 raft_consensus.cc:686] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 2 LEADER]: Becoming Leader. State: Replica: 713b76caded84f269397165703739870, State: Running, Role: LEADER
I20240503 08:00:06.070556 12300 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:06.072180 9335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08511a1b1eab4da9b810283d12cccee3" candidate_uuid: "713b76caded84f269397165703739870" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0394d429c78744aca1220af8f39b11c1"
I20240503 08:00:06.072149 6164 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 reported cstate change: term changed from 1 to 2, leader changed from 253dd189c3c542248402cba222394f43 (127.0.99.129) to 713b76caded84f269397165703739870 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "713b76caded84f269397165703739870" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:06.072283 9335 raft_consensus.cc:3037] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:06.073125 9335 raft_consensus.cc:2454] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713b76caded84f269397165703739870 in term 2.
I20240503 08:00:06.083616 6568 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.083992 12421 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.118275 6304 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.118767 12300 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.132397 8165 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.132735 12300 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.135272 6437 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.136099 13075 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.142061 8427 consensus_queue.cc:237] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } }
I20240503 08:00:06.143003 8165 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:06.143049 6304 raft_consensus.cc:1261] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:06.143342 12300 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:06.143657 12300 consensus_queue.cc:1035] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:06.145251 13140 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:06.145737 6304 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:06.145855 8165 raft_consensus.cc:2932] T d9e91e1b70504ac695eae61f72d3c149 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } }
I20240503 08:00:06.146852 6151 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9e91e1b70504ac695eae61f72d3c149 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:06.148361 6157 catalog_manager.cc:5484] T d9e91e1b70504ac695eae61f72d3c149 P 713b76caded84f269397165703739870 reported cstate change: config changed from index 8 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "713b76caded84f269397165703739870" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:06.151242 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e91e1b70504ac695eae61f72d3c149 with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34676
I20240503 08:00:06.151541 13147 tablet_replica.cc:330] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 08:00:06.151664 13147 raft_consensus.cc:2229] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:06.151767 13147 raft_consensus.cc:2258] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:06.152467 13147 ts_tablet_manager.cc:1875] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:06.154426 13147 ts_tablet_manager.cc:1888] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:00:06.154528 13147 log.cc:1196] T d9e91e1b70504ac695eae61f72d3c149 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d9e91e1b70504ac695eae61f72d3c149
I20240503 08:00:06.155081 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet d9e91e1b70504ac695eae61f72d3c149 (table rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]) successfully deleted
I20240503 08:00:06.160192 13158 ts_tablet_manager.cc:902] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351)
I20240503 08:00:06.160722 13158 tablet_copy_client.cc:308] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:34351
I20240503 08:00:06.162010 8165 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.162499 12421 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.166225 8296 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 2 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.166607 6456 tablet_copy_service.cc:140] P 02fe52b383f3498ebbbe4542c8cc12fb: Received BeginTabletCopySession request for tablet 4925e30c2fb74934bbd5eb7415193c96 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:39917)
I20240503 08:00:06.166687 6456 tablet_copy_service.cc:161] P 02fe52b383f3498ebbbe4542c8cc12fb: Beginning new tablet copy session on tablet 4925e30c2fb74934bbd5eb7415193c96 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:39917: session id = 0394d429c78744aca1220af8f39b11c1-4925e30c2fb74934bbd5eb7415193c96
I20240503 08:00:06.167225 12723 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.167431 6456 tablet_copy_source_session.cc:205] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:06.169179 13158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4925e30c2fb74934bbd5eb7415193c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.171046 13158 tablet_copy_client.cc:783] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:06.171169 13158 tablet_copy_client.cc:647] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:06.173736 13161 tablet_copy_client.cc:1002] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:06.199599 8165 consensus_queue.cc:237] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } }
I20240503 08:00:06.200826 8033 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:06.200881 6568 raft_consensus.cc:1261] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Refusing update from remote peer 66f9a693c3b3409dbcce8f3ee500bf5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:06.201272 13135 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:06.201450 12317 consensus_queue.cc:1035] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:06.202967 6698 consensus_queue.cc:237] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:06.204039 13135 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:06.204320 6437 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:06.204595 6568 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:06.205256 12517 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:06.205523 8296 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:06.206246 6157 catalog_manager.cc:5484] T d17b7dcf55364209bd26129488dd0184 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 8 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:06.206425 12517 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:06.206974 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d17b7dcf55364209bd26129488dd0184 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:06.207136 8033 raft_consensus.cc:2932] T d17b7dcf55364209bd26129488dd0184 P 61a9a0a13deb4d72b73f8623f6576766 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } }
I20240503 08:00:06.209008 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b7dcf55364209bd26129488dd0184 with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34676
I20240503 08:00:06.209316 13147 tablet_replica.cc:330] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 08:00:06.209452 13147 raft_consensus.cc:2229] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:06.209530 13147 raft_consensus.cc:2258] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:06.210387 13147 ts_tablet_manager.cc:1875] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:06.210379 12519 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:06.210820 8296 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:06.212330 13147 ts_tablet_manager.cc:1888] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:00:06.212400 13147 log.cc:1196] T d17b7dcf55364209bd26129488dd0184 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d17b7dcf55364209bd26129488dd0184
I20240503 08:00:06.213454 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet d17b7dcf55364209bd26129488dd0184 (table rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]) successfully deleted
I20240503 08:00:06.213855 6155 catalog_manager.cc:5484] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee reported cstate change: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
W20240503 08:00:06.216408 6633 consensus_peers.cc:485] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 551b0bab14bb4bb492885bada3e031a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:06.216552 6437 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:06.511265 9335 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.511624 12300 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.615012 6305 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.615453 12300 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.618124 6698 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:00:06.618826 13140 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:06.655404 6437 consensus_queue.cc:237] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } }
I20240503 08:00:06.656543 8296 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 2 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:06.656543 8165 raft_consensus.cc:1261] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:06.656945 12723 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:06.657210 13164 consensus_queue.cc:1035] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:06.658766 12421 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:06.659164 8295 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P cae9627a7600448faa8fcb7989b307ee [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:06.659196 8165 raft_consensus.cc:2932] T 67b9602786f8422d962885dda25c6c85 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } }
I20240503 08:00:06.660080 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67b9602786f8422d962885dda25c6c85 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:06.661631 6155 catalog_manager.cc:5484] T 67b9602786f8422d962885dda25c6c85 P 02fe52b383f3498ebbbe4542c8cc12fb reported cstate change: config changed from index 8 to 10, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:06.663865 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b9602786f8422d962885dda25c6c85 with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41936
I20240503 08:00:06.664248 13304 tablet_replica.cc:330] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 08:00:06.664511 13304 raft_consensus.cc:2229] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:06.664628 13304 raft_consensus.cc:2258] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:06.665176 13304 ts_tablet_manager.cc:1875] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:06.666914 13304 ts_tablet_manager.cc:1888] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:00:06.666973 13304 log.cc:1196] T 67b9602786f8422d962885dda25c6c85 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/67b9602786f8422d962885dda25c6c85
I20240503 08:00:06.667435 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet 67b9602786f8422d962885dda25c6c85 (table rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]) successfully deleted
I20240503 08:00:06.779073 13360 ts_tablet_manager.cc:902] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367)
I20240503 08:00:06.779481 13360 tablet_copy_client.cc:308] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:43367
I20240503 08:00:06.782609 6719 tablet_copy_service.cc:140] P e9976bc168264eaeba1d180fb38b5113: Received BeginTabletCopySession request for tablet 551b0bab14bb4bb492885bada3e031a0 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:35843)
I20240503 08:00:06.782711 6719 tablet_copy_service.cc:161] P e9976bc168264eaeba1d180fb38b5113: Beginning new tablet copy session on tablet 551b0bab14bb4bb492885bada3e031a0 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:35843: session id = 0394d429c78744aca1220af8f39b11c1-551b0bab14bb4bb492885bada3e031a0
I20240503 08:00:06.783612 6719 tablet_copy_source_session.cc:205] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:06.784396 13360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551b0bab14bb4bb492885bada3e031a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:06.786201 13360 tablet_copy_client.cc:783] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:06.786347 13360 tablet_copy_client.cc:647] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:06.787305 13363 tablet_copy_client.cc:1002] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:07.006718 8427 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:07.007725 6698 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:07.007828 9335 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:00:07.008067 13075 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:07.008239 13140 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:07.009718 12300 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:07.010167 9335 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:07.010262 6698 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:07.011737 6151 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08511a1b1eab4da9b810283d12cccee3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:07.012308 6157 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 reported cstate change: config changed from index 8 to 10, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "713b76caded84f269397165703739870" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:07.014447 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 08511a1b1eab4da9b810283d12cccee3 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35598
I20240503 08:00:07.014803 13393 tablet_replica.cc:330] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 08:00:07.014938 13393 raft_consensus.cc:2229] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:07.015044 13393 raft_consensus.cc:2258] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:07.015736 13393 ts_tablet_manager.cc:1875] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:07.017589 13393 ts_tablet_manager.cc:1888] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:00:07.017649 13393 log.cc:1196] T 08511a1b1eab4da9b810283d12cccee3 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/08511a1b1eab4da9b810283d12cccee3
I20240503 08:00:07.018070 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 08511a1b1eab4da9b810283d12cccee3 (table rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]) successfully deleted
I20240503 08:00:07.094913 12961 tablet_copy_client.cc:523] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:07.096218 12961 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:07.113479 12961 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:07.113683 12961 tablet_bootstrap.cc:492] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:07.113806 12961 ts_tablet_manager.cc:1367] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240503 08:00:07.114151 12961 raft_consensus.cc:348] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:07.114265 12961 raft_consensus.cc:729] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:07.114341 12961 consensus_queue.cc:260] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:07.114567 12961 ts_tablet_manager.cc:1398] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:07.115067 6326 tablet_copy_service.cc:342] P 253dd189c3c542248402cba222394f43: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-e03d7158b64f434a8871f009864a0b72 received from {username='slave'} at 127.0.99.137:38407
I20240503 08:00:07.115135 6326 tablet_copy_service.cc:433] P 253dd189c3c542248402cba222394f43: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-e03d7158b64f434a8871f009864a0b72 on tablet e03d7158b64f434a8871f009864a0b72 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:07.228778 6568 consensus_queue.cc:237] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:07.230022 6437 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:00:07.230249 6502 consensus_peers.cc:485] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 -> Peer 0394d429c78744aca1220af8f39b11c1 (127.0.99.137:39867): Couldn't send request to peer 0394d429c78744aca1220af8f39b11c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9023b7fb87a4adaa91995b883df8826. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:07.230401 8165 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:07.230444 13367 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:07.230760 13366 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:07.232278 13367 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:07.233114 8165 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:07.233582 6437 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:07.235304 6155 catalog_manager.cc:5484] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 6 to 7, NON_VOTER 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } } }
I20240503 08:00:07.491133 13128 tablet_copy_client.cc:523] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:07.492328 13128 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:07.510870 13128 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:07.511090 13128 tablet_bootstrap.cc:492] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:07.511246 13128 ts_tablet_manager.cc:1367] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.005s
I20240503 08:00:07.511639 13128 raft_consensus.cc:348] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:07.511740 13128 raft_consensus.cc:729] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:07.511826 13128 consensus_queue.cc:260] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:07.512058 13128 ts_tablet_manager.cc:1398] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:07.512522 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-7d608390b027432f8a3f89130c051759 received from {username='slave'} at 127.0.99.137:42151
I20240503 08:00:07.512598 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-7d608390b027432f8a3f89130c051759 on tablet 7d608390b027432f8a3f89130c051759 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:07.612171 9335 raft_consensus.cc:1203] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:07.674013 13158 tablet_copy_client.cc:523] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:07.675518 13158 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:07.698917 13158 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:07.699256 13158 tablet_bootstrap.cc:492] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:07.699457 13158 ts_tablet_manager.cc:1367] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.024s user 0.025s sys 0.000s
I20240503 08:00:07.699923 13158 raft_consensus.cc:348] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:07.700052 13158 raft_consensus.cc:729] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:07.700157 13158 consensus_queue.cc:260] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:07.700415 13158 ts_tablet_manager.cc:1398] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:07.700891 6456 tablet_copy_service.cc:342] P 02fe52b383f3498ebbbe4542c8cc12fb: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-4925e30c2fb74934bbd5eb7415193c96 received from {username='slave'} at 127.0.99.137:39917
I20240503 08:00:07.700960 6456 tablet_copy_service.cc:433] P 02fe52b383f3498ebbbe4542c8cc12fb: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-4925e30c2fb74934bbd5eb7415193c96 on tablet 4925e30c2fb74934bbd5eb7415193c96 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:07.809082 13158 ts_tablet_manager.cc:902] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: Initiating tablet copy from peer 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099)
I20240503 08:00:07.809505 13158 tablet_copy_client.cc:308] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34099
I20240503 08:00:07.809918 6588 tablet_copy_service.cc:140] P 34abd76ae3764e7f9641eaddaffdd5a5: Received BeginTabletCopySession request for tablet a9023b7fb87a4adaa91995b883df8826 from peer 0394d429c78744aca1220af8f39b11c1 ({username='slave'} at 127.0.99.137:42151)
I20240503 08:00:07.810014 6588 tablet_copy_service.cc:161] P 34abd76ae3764e7f9641eaddaffdd5a5: Beginning new tablet copy session on tablet a9023b7fb87a4adaa91995b883df8826 from peer 0394d429c78744aca1220af8f39b11c1 at {username='slave'} at 127.0.99.137:42151: session id = 0394d429c78744aca1220af8f39b11c1-a9023b7fb87a4adaa91995b883df8826
I20240503 08:00:07.810684 6588 tablet_copy_source_session.cc:205] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:07.811443 13158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9023b7fb87a4adaa91995b883df8826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:07.813225 13158 tablet_copy_client.cc:783] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:07.813359 13158 tablet_copy_client.cc:647] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:07.814368 13510 tablet_copy_client.cc:1002] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:07.997421 12008 raft_consensus.cc:1050] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:07.997866 12008 consensus_queue.cc:237] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:07.999166 9335 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:07.999176 8427 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:07.999351 6568 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:07.999518 13536 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:07.999714 13538 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:07.999926 13539 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:08.002173 13536 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.002573 6568 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.002568 9335 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.004261 8427 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.004869 6163 catalog_manager.cc:5484] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 reported cstate change: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.008466 6305 consensus_queue.cc:237] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:08.009497 9335 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:08.009557 8428 raft_consensus.cc:1261] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Refusing update from remote peer 253dd189c3c542248402cba222394f43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:08.009982 12008 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:08.010174 13538 consensus_queue.cc:1035] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:08.011648 13538 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 253dd189c3c542248402cba222394f43 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.012056 9335 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.012066 8428 raft_consensus.cc:2932] T e03d7158b64f434a8871f009864a0b72 P 713b76caded84f269397165703739870 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.013919 6164 catalog_manager.cc:5484] T e03d7158b64f434a8871f009864a0b72 P 0394d429c78744aca1220af8f39b11c1 reported cstate change: config changed from index 8 to 9, VOTER 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "253dd189c3c542248402cba222394f43" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.014417 6149 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e03d7158b64f434a8871f009864a0b72 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:08.015919 6546 tablet_service.cc:1514] Processing DeleteTablet for tablet e03d7158b64f434a8871f009864a0b72 with delete_type TABLET_DATA_TOMBSTONED (TS 34abd76ae3764e7f9641eaddaffdd5a5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41936
I20240503 08:00:08.016218 13546 tablet_replica.cc:330] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5: stopping tablet replica
I20240503 08:00:08.016402 13546 raft_consensus.cc:2229] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:08.016640 13546 raft_consensus.cc:2258] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:08.017237 13546 ts_tablet_manager.cc:1875] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:08.018836 13546 ts_tablet_manager.cc:1888] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:08.018893 13546 log.cc:1196] T e03d7158b64f434a8871f009864a0b72 P 34abd76ae3764e7f9641eaddaffdd5a5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e03d7158b64f434a8871f009864a0b72
I20240503 08:00:08.019276 6150 catalog_manager.cc:4830] TS 34abd76ae3764e7f9641eaddaffdd5a5 (127.0.99.131:34099): tablet e03d7158b64f434a8871f009864a0b72 (table rebalance_test_table_5 [id=2f7653599e75445d89ea3a1b7a742734]) successfully deleted
I20240503 08:00:08.088107 9335 raft_consensus.cc:1203] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:08.180251 9335 raft_consensus.cc:1203] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:08.287595 13360 tablet_copy_client.cc:523] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:08.288769 13360 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:08.306847 13360 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:08.317286 13360 tablet_bootstrap.cc:492] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:08.317456 13360 ts_tablet_manager.cc:1367] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.000s
I20240503 08:00:08.317823 13360 raft_consensus.cc:348] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:08.317945 13360 raft_consensus.cc:729] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:08.318032 13360 consensus_queue.cc:260] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:08.318269 13360 ts_tablet_manager.cc:1398] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:08.318745 6719 tablet_copy_service.cc:342] P e9976bc168264eaeba1d180fb38b5113: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-551b0bab14bb4bb492885bada3e031a0 received from {username='slave'} at 127.0.99.137:35843
I20240503 08:00:08.318818 6719 tablet_copy_service.cc:433] P e9976bc168264eaeba1d180fb38b5113: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-551b0bab14bb4bb492885bada3e031a0 on tablet 551b0bab14bb4bb492885bada3e031a0 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:08.612689 13593 raft_consensus.cc:1050] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:08.613166 13593 consensus_queue.cc:237] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:08.614558 8033 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:08.614699 9335 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:08.614846 6305 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:08.615064 13366 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:08.615628 13592 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:08.615808 13593 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:08.617826 13592 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.618227 6305 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.620587 8033 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.620609 6157 catalog_manager.cc:5484] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:08.621253 9335 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.626420 6568 consensus_queue.cc:237] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:08.627440 8033 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:08.627487 9335 raft_consensus.cc:1261] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:08.627769 13593 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:08.627943 13366 consensus_queue.cc:1035] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:08.629560 13621 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.629937 8033 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 61a9a0a13deb4d72b73f8623f6576766 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.630470 9335 raft_consensus.cc:2932] T 7d608390b027432f8a3f89130c051759 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.631153 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d608390b027432f8a3f89130c051759 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:08.632040 6155 catalog_manager.cc:5484] T 7d608390b027432f8a3f89130c051759 P 34abd76ae3764e7f9641eaddaffdd5a5 reported cstate change: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61a9a0a13deb4d72b73f8623f6576766" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:08.634804 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d608390b027432f8a3f89130c051759 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35598
I20240503 08:00:08.635890 13638 tablet_replica.cc:330] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 08:00:08.636050 13638 raft_consensus.cc:2229] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:08.636240 13638 raft_consensus.cc:2258] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:08.636783 13638 ts_tablet_manager.cc:1875] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:08.638731 13638 ts_tablet_manager.cc:1888] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:08.638794 13638 log.cc:1196] T 7d608390b027432f8a3f89130c051759 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d608390b027432f8a3f89130c051759
I20240503 08:00:08.639212 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 7d608390b027432f8a3f89130c051759 (table rebalance_test_table_9 [id=1afb8461f1784a0faeb87583c936217d]) successfully deleted
I20240503 08:00:08.733604 13660 raft_consensus.cc:1050] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:08.734006 13660 consensus_queue.cc:237] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:08.735170 9335 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:08.735311 8295 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:08.735405 6305 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:08.735611 13659 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:08.735780 13660 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:08.735970 13662 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:08.737756 13660 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.738212 9335 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.738464 8295 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.740687 6155 catalog_manager.cc:5484] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 reported cstate change: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.741197 6305 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "253dd189c3c542248402cba222394f43" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 38721 } attrs { replace: true } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.743568 6437 consensus_queue.cc:237] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:08.744937 9335 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:08.744941 8295 raft_consensus.cc:1261] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer 02fe52b383f3498ebbbe4542c8cc12fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:08.745275 13662 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:08.745430 13659 consensus_queue.cc:1035] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:08.746716 13660 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.747167 8295 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.747167 9335 raft_consensus.cc:2932] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.748795 6148 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4925e30c2fb74934bbd5eb7415193c96 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:08.749300 6164 catalog_manager.cc:5484] T 4925e30c2fb74934bbd5eb7415193c96 P 0394d429c78744aca1220af8f39b11c1 reported cstate change: config changed from index 8 to 9, VOTER 253dd189c3c542248402cba222394f43 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:08.751302 6284 tablet_service.cc:1514] Processing DeleteTablet for tablet 4925e30c2fb74934bbd5eb7415193c96 with delete_type TABLET_DATA_TOMBSTONED (TS 253dd189c3c542248402cba222394f43 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35598
I20240503 08:00:08.751506 13638 tablet_replica.cc:330] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43: stopping tablet replica
I20240503 08:00:08.751636 13638 raft_consensus.cc:2229] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:08.751801 13638 raft_consensus.cc:2258] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:08.752398 13638 ts_tablet_manager.cc:1875] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:08.754011 13638 ts_tablet_manager.cc:1888] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:08.754069 13638 log.cc:1196] T 4925e30c2fb74934bbd5eb7415193c96 P 253dd189c3c542248402cba222394f43: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4925e30c2fb74934bbd5eb7415193c96
I20240503 08:00:08.754519 6149 catalog_manager.cc:4830] TS 253dd189c3c542248402cba222394f43 (127.0.99.129:38721): tablet 4925e30c2fb74934bbd5eb7415193c96 (table rebalance_test_table_2 [id=b27a495250cd4256a051f042cd8fafd9]) successfully deleted
I20240503 08:00:08.846622 9335 raft_consensus.cc:1203] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:09.314639 13158 tablet_copy_client.cc:523] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:09.315964 13158 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: Bootstrap starting.
I20240503 08:00:09.333873 13158 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:09.334107 13158 tablet_bootstrap.cc:492] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: Bootstrap complete.
I20240503 08:00:09.334261 13158 ts_tablet_manager.cc:1367] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240503 08:00:09.334622 13158 raft_consensus.cc:348] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:09.334726 13158 raft_consensus.cc:729] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0394d429c78744aca1220af8f39b11c1, State: Initialized, Role: LEARNER
I20240503 08:00:09.334806 13158 consensus_queue.cc:260] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: true } }
I20240503 08:00:09.335081 13158 ts_tablet_manager.cc:1398] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:09.335600 6588 tablet_copy_service.cc:342] P 34abd76ae3764e7f9641eaddaffdd5a5: Request end of tablet copy session 0394d429c78744aca1220af8f39b11c1-a9023b7fb87a4adaa91995b883df8826 received from {username='slave'} at 127.0.99.137:42151
I20240503 08:00:09.335680 6588 tablet_copy_service.cc:433] P 34abd76ae3764e7f9641eaddaffdd5a5: ending tablet copy session 0394d429c78744aca1220af8f39b11c1-a9023b7fb87a4adaa91995b883df8826 on tablet a9023b7fb87a4adaa91995b883df8826 with peer 0394d429c78744aca1220af8f39b11c1
I20240503 08:00:09.461652 13714 raft_consensus.cc:1050] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:09.462034 13714 consensus_queue.cc:237] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:09.463248 8295 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:09.463307 9336 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:09.463423 6437 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:09.463647 13713 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:09.463871 13716 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:09.464035 13714 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:09.465795 13716 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:09.466214 6437 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:09.466527 9336 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:09.468830 6155 catalog_manager.cc:5484] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:09.469923 8295 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:09.472839 6698 consensus_queue.cc:237] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:09.473906 9336 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:09.474323 13716 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:09.474578 8295 raft_consensus.cc:1261] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Refusing update from remote peer e9976bc168264eaeba1d180fb38b5113: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:09.474920 13716 consensus_queue.cc:1035] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:09.475787 13714 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:09.476270 9336 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:09.476979 8295 raft_consensus.cc:2932] T 551b0bab14bb4bb492885bada3e031a0 P cae9627a7600448faa8fcb7989b307ee [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:09.477260 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 551b0bab14bb4bb492885bada3e031a0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:09.478260 6164 catalog_manager.cc:5484] T 551b0bab14bb4bb492885bada3e031a0 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "e9976bc168264eaeba1d180fb38b5113" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:09.480464 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet 551b0bab14bb4bb492885bada3e031a0 with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34676
I20240503 08:00:09.480803 13724 tablet_replica.cc:330] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 08:00:09.480921 13724 raft_consensus.cc:2229] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:09.481140 13724 raft_consensus.cc:2258] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:09.481704 13724 ts_tablet_manager.cc:1875] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:09.483384 13724 ts_tablet_manager.cc:1888] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:09.483441 13724 log.cc:1196] T 551b0bab14bb4bb492885bada3e031a0 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/551b0bab14bb4bb492885bada3e031a0
I20240503 08:00:09.483897 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet 551b0bab14bb4bb492885bada3e031a0 (table rebalance_test_table_6 [id=35f11d4cc363465e822b523d1ddc7ee0]) successfully deleted
I20240503 08:00:09.706843 9335 raft_consensus.cc:1203] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:00:10.207365 13757 raft_consensus.cc:1050] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5: attempting to promote NON_VOTER 0394d429c78744aca1220af8f39b11c1 to VOTER
I20240503 08:00:10.207787 13757 consensus_queue.cc:237] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:10.209190 9335 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1 [term 1 LEARNER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:10.209209 8165 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:10.209296 6437 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:10.209638 13756 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:10.209869 13757 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:10.210034 13366 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:10.211861 13756 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:10.212339 9335 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:10.212597 8165 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:10.214149 6437 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:10.215502 6163 catalog_manager.cc:5484] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 7 to 8, 0394d429c78744aca1220af8f39b11c1 (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "02fe52b383f3498ebbbe4542c8cc12fb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 34351 } attrs { replace: true } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:10.219085 6568 consensus_queue.cc:237] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } }
I20240503 08:00:10.220156 8165 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:10.220322 9335 raft_consensus.cc:1261] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Refusing update from remote peer 34abd76ae3764e7f9641eaddaffdd5a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:10.220520 13366 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:10.220717 13756 consensus_queue.cc:1035] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:10.222436 13759 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 34abd76ae3764e7f9641eaddaffdd5a5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:10.222754 8165 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:10.222848 9335 raft_consensus.cc:2932] T a9023b7fb87a4adaa91995b883df8826 P 0394d429c78744aca1220af8f39b11c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:10.223623 6150 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a9023b7fb87a4adaa91995b883df8826 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:10.224946 6157 catalog_manager.cc:5484] T a9023b7fb87a4adaa91995b883df8826 P 66f9a693c3b3409dbcce8f3ee500bf5e reported cstate change: config changed from index 8 to 9, VOTER 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "34abd76ae3764e7f9641eaddaffdd5a5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34099 } } peers { permanent_uuid: "66f9a693c3b3409dbcce8f3ee500bf5e" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 42851 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } }
I20240503 08:00:10.227571 6415 tablet_service.cc:1514] Processing DeleteTablet for tablet a9023b7fb87a4adaa91995b883df8826 with delete_type TABLET_DATA_TOMBSTONED (TS 02fe52b383f3498ebbbe4542c8cc12fb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34676
I20240503 08:00:10.227891 13767 tablet_replica.cc:330] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb: stopping tablet replica
I20240503 08:00:10.228019 13767 raft_consensus.cc:2229] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:10.228192 13767 raft_consensus.cc:2258] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:10.228804 13767 ts_tablet_manager.cc:1875] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:10.230504 13767 ts_tablet_manager.cc:1888] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:00:10.230571 13767 log.cc:1196] T a9023b7fb87a4adaa91995b883df8826 P 02fe52b383f3498ebbbe4542c8cc12fb: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a9023b7fb87a4adaa91995b883df8826
I20240503 08:00:10.231050 6148 catalog_manager.cc:4830] TS 02fe52b383f3498ebbbe4542c8cc12fb (127.0.99.130:34351): tablet a9023b7fb87a4adaa91995b883df8826 (table rebalance_test_table_4 [id=c8c192252f264f9eb147cdcc832fc891]) successfully deleted
I20240503 08:00:10.423593 9316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.423945 6415 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:10.424427 6284 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:10.425256 6546 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:10.432363 8407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:10.436764 8014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:10.437211 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:10.462975 8145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:10.470969 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:10.503268 8428 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:10.504801 9335 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:00:10.504854 6698 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 08:00:10.505242 13075 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:10.505553 13075 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:10.506978 13075 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 08:00:10.507265 6698 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 08:00:10.509817 6155 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 reported cstate change: config changed from index 10 to 11, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "713b76caded84f269397165703739870" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
W20240503 08:00:10.510272 8362 consensus_peers.cc:485] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 -> Peer cae9627a7600448faa8fcb7989b307ee (127.0.99.135:35381): Couldn't send request to peer cae9627a7600448faa8fcb7989b307ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08511a1b1eab4da9b810283d12cccee3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:10.512148 9335 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER cae9627a7600448faa8fcb7989b307ee (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } } }
I20240503 08:00:11.096958 13848 ts_tablet_manager.cc:902] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: Initiating tablet copy from peer 713b76caded84f269397165703739870 (127.0.99.136:38867)
I20240503 08:00:11.097445 13848 tablet_copy_client.cc:308] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:38867
I20240503 08:00:11.101114 8448 tablet_copy_service.cc:140] P 713b76caded84f269397165703739870: Received BeginTabletCopySession request for tablet 08511a1b1eab4da9b810283d12cccee3 from peer cae9627a7600448faa8fcb7989b307ee ({username='slave'} at 127.0.99.135:52543)
I20240503 08:00:11.101241 8448 tablet_copy_service.cc:161] P 713b76caded84f269397165703739870: Beginning new tablet copy session on tablet 08511a1b1eab4da9b810283d12cccee3 from peer cae9627a7600448faa8fcb7989b307ee at {username='slave'} at 127.0.99.135:52543: session id = cae9627a7600448faa8fcb7989b307ee-08511a1b1eab4da9b810283d12cccee3
I20240503 08:00:11.102370 8448 tablet_copy_source_session.cc:205] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:11.103394 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08511a1b1eab4da9b810283d12cccee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:11.105248 13848 tablet_copy_client.cc:783] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:11.105401 13848 tablet_copy_client.cc:647] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:11.106679 13852 tablet_copy_client.cc:1002] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Injecting latency into file download: 1500
I20240503 08:00:12.606978 13848 tablet_copy_client.cc:523] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:12.608410 13848 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: Bootstrap starting.
I20240503 08:00:12.626936 13848 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:12.627148 13848 tablet_bootstrap.cc:492] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: Bootstrap complete.
I20240503 08:00:12.627272 13848 ts_tablet_manager.cc:1367] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240503 08:00:12.627594 13848 raft_consensus.cc:348] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:12.627698 13848 raft_consensus.cc:729] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cae9627a7600448faa8fcb7989b307ee, State: Initialized, Role: LEARNER
I20240503 08:00:12.627768 13848 consensus_queue.cc:260] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: true } }
I20240503 08:00:12.628054 13848 ts_tablet_manager.cc:1398] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:00:12.628679 8448 tablet_copy_service.cc:342] P 713b76caded84f269397165703739870: Request end of tablet copy session cae9627a7600448faa8fcb7989b307ee-08511a1b1eab4da9b810283d12cccee3 received from {username='slave'} at 127.0.99.135:52543
I20240503 08:00:12.628769 8448 tablet_copy_service.cc:433] P 713b76caded84f269397165703739870: ending tablet copy session cae9627a7600448faa8fcb7989b307ee-08511a1b1eab4da9b810283d12cccee3 on tablet 08511a1b1eab4da9b810283d12cccee3 with peer cae9627a7600448faa8fcb7989b307ee
I20240503 08:00:13.154332 8295 raft_consensus.cc:1203] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240503 08:00:13.604636 13872 raft_consensus.cc:1050] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870: attempting to promote NON_VOTER cae9627a7600448faa8fcb7989b307ee to VOTER
I20240503 08:00:13.605165 13872 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:13.606598 6698 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:00:13.606918 8295 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee [term 2 LEARNER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:00:13.607021 9336 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 08:00:13.607053 13927 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:13.607308 13928 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:13.607499 13930 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:13.609087 13927 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:13.609563 8295 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:13.609577 9336 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:13.611426 6698 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:13.611639 6155 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 reported cstate change: config changed from index 11 to 12, cae9627a7600448faa8fcb7989b307ee (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "713b76caded84f269397165703739870" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9976bc168264eaeba1d180fb38b5113" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43367 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:13.615017 8428 consensus_queue.cc:237] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } }
I20240503 08:00:13.615990 8295 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:00:13.616050 9336 raft_consensus.cc:1261] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Refusing update from remote peer 713b76caded84f269397165703739870: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 08:00:13.616403 13930 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:00:13.616559 13927 consensus_queue.cc:1035] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 08:00:13.617831 13872 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:13.618909 6151 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08511a1b1eab4da9b810283d12cccee3 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:13.619217 9336 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P 0394d429c78744aca1220af8f39b11c1 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:13.619264 8295 raft_consensus.cc:2932] T 08511a1b1eab4da9b810283d12cccee3 P cae9627a7600448faa8fcb7989b307ee [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } } }
I20240503 08:00:13.620759 6157 catalog_manager.cc:5484] T 08511a1b1eab4da9b810283d12cccee3 P 713b76caded84f269397165703739870 reported cstate change: config changed from index 12 to 13, VOTER e9976bc168264eaeba1d180fb38b5113 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "713b76caded84f269397165703739870" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "713b76caded84f269397165703739870" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 38867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0394d429c78744aca1220af8f39b11c1" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 39867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae9627a7600448faa8fcb7989b307ee" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 35381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:13.622954 6677 tablet_service.cc:1514] Processing DeleteTablet for tablet 08511a1b1eab4da9b810283d12cccee3 with delete_type TABLET_DATA_TOMBSTONED (TS e9976bc168264eaeba1d180fb38b5113 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:45994
I20240503 08:00:13.623626 13940 tablet_replica.cc:330] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113: stopping tablet replica
I20240503 08:00:13.623795 13940 raft_consensus.cc:2229] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:13.624019 13940 raft_consensus.cc:2258] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:13.624577 13940 ts_tablet_manager.cc:1875] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:13.626266 13940 ts_tablet_manager.cc:1888] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240503 08:00:13.626329 13940 log.cc:1196] T 08511a1b1eab4da9b810283d12cccee3 P e9976bc168264eaeba1d180fb38b5113: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/08511a1b1eab4da9b810283d12cccee3
I20240503 08:00:13.626721 6150 catalog_manager.cc:4830] TS e9976bc168264eaeba1d180fb38b5113 (127.0.99.132:43367): tablet 08511a1b1eab4da9b810283d12cccee3 (table rebalance_test_table_1 [id=1e210154abf34473a857a0b342746f60]) successfully deleted
I20240503 08:00:13.872803 9316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.874935 6284 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:13.878386 6415 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:13.879513 6546 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:13.889331 8407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:13.900542 8145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:13.909494 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:13.922700 8014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:13.924973 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.015039 6284 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:14.018505 6546 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.021775 8407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:14.023237 8145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:14.026300 9316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.032665 8014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:14.050241 6415 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.051928 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.062355 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.279897 6546 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.281040 6415 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.283179 9316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.293440 6284 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:14.297437 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.306833 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.309121 8407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:14.309351 8014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:14.313136 8145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:14.424211 6415 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.426224 8014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:14.433789 8145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:14.434136 9316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.434592 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.439937 6284 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:14.452304 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.461980 8407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:14.475824 6546 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.600844 9316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.602170 6415 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.604542 6546 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.606688 8014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:14.613859 6284 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:14.618548 8407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:14.619341 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.645998 8145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:14.647482 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.762079 6546 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.762881 6284 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:14.763962 8145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:00:14.764822 8014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:14.765682 8276 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.766258 9316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:14.786239 6415 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.786767 6677 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:14.815172 8407 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
56c0b66969e4477fbb9ae5d329115122 | 127.0.99.190:42987 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34965 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
02fe52b383f3498ebbbe4542c8cc12fb | 127.0.99.130:34351 | HEALTHY | <none> | 6 | 0
0394d429c78744aca1220af8f39b11c1 | 127.0.99.137:39867 | HEALTHY | <none> | 0 | 0
253dd189c3c542248402cba222394f43 | 127.0.99.129:38721 | HEALTHY | <none> | 7 | 0
34abd76ae3764e7f9641eaddaffdd5a5 | 127.0.99.131:34099 | HEALTHY | <none> | 6 | 0
61a9a0a13deb4d72b73f8623f6576766 | 127.0.99.133:41285 | HEALTHY | <none> | 1 | 0
66f9a693c3b3409dbcce8f3ee500bf5e | 127.0.99.134:42851 | HEALTHY | <none> | 2 | 0
713b76caded84f269397165703739870 | 127.0.99.136:38867 | HEALTHY | <none> | 2 | 0
cae9627a7600448faa8fcb7989b307ee | 127.0.99.135:35381 | HEALTHY | <none> | 0 | 0
e9976bc168264eaeba1d180fb38b5113 | 127.0.99.132:43367 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 9
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
---------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:38721
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:34351
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:34099
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:43367
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:41285
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:42851
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:35381
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:38867
local_ip_for_outbound_sockets | 127.0.99.137 | experimental | 127.0.99.137:39867
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:38721
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:34351
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:34099
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:43367
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:41285
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:42851
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:35381
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:38867
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.0.99.137:39867
tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34965 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 10
Tablets | 30
Replicas | 90
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:00:14.838727 398 log_verifier.cc:126] Checking tablet 08511a1b1eab4da9b810283d12cccee3
I20240503 08:00:14.864495 398 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 08511a1b1eab4da9b810283d12cccee3
I20240503 08:00:14.864559 398 log_verifier.cc:126] Checking tablet 2f40fae73f8f4c63b2d656ac880799e2
I20240503 08:00:14.888851 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 2f40fae73f8f4c63b2d656ac880799e2
I20240503 08:00:14.888916 398 log_verifier.cc:126] Checking tablet 2fb6518dced248a88edde5cbbb25350d
I20240503 08:00:14.912791 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 2fb6518dced248a88edde5cbbb25350d
I20240503 08:00:14.912853 398 log_verifier.cc:126] Checking tablet 37c8748a1c9d42b593b2381a1a870d2e
I20240503 08:00:14.936453 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 37c8748a1c9d42b593b2381a1a870d2e
I20240503 08:00:14.936522 398 log_verifier.cc:126] Checking tablet 387f6b1606b94e11897372c0557870e4
I20240503 08:00:14.960877 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 387f6b1606b94e11897372c0557870e4
I20240503 08:00:14.960940 398 log_verifier.cc:126] Checking tablet 3d8a5e7a21684a4782d43c5a1131f652
I20240503 08:00:14.984143 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3d8a5e7a21684a4782d43c5a1131f652
I20240503 08:00:14.984205 398 log_verifier.cc:126] Checking tablet 4418f348094548daa128685753415583
I20240503 08:00:15.006683 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4418f348094548daa128685753415583
I20240503 08:00:15.006745 398 log_verifier.cc:126] Checking tablet 4925e30c2fb74934bbd5eb7415193c96
I20240503 08:00:15.030320 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 4925e30c2fb74934bbd5eb7415193c96
I20240503 08:00:15.030390 398 log_verifier.cc:126] Checking tablet 551b0bab14bb4bb492885bada3e031a0
I20240503 08:00:15.055435 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 551b0bab14bb4bb492885bada3e031a0
I20240503 08:00:15.055500 398 log_verifier.cc:126] Checking tablet 64eb15b0505a4649a0043be00250c299
I20240503 08:00:15.079473 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 64eb15b0505a4649a0043be00250c299
I20240503 08:00:15.079535 398 log_verifier.cc:126] Checking tablet 67b9602786f8422d962885dda25c6c85
I20240503 08:00:15.104987 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 67b9602786f8422d962885dda25c6c85
I20240503 08:00:15.105047 398 log_verifier.cc:126] Checking tablet 7d608390b027432f8a3f89130c051759
I20240503 08:00:15.129317 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 7d608390b027432f8a3f89130c051759
I20240503 08:00:15.129379 398 log_verifier.cc:126] Checking tablet 82f6c17706cc4a40827501ecb7c3ebce
I20240503 08:00:15.152487 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 82f6c17706cc4a40827501ecb7c3ebce
I20240503 08:00:15.152549 398 log_verifier.cc:126] Checking tablet 8625e8b653624ea6acf6e04c3fd352a4
I20240503 08:00:15.177021 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 8625e8b653624ea6acf6e04c3fd352a4
I20240503 08:00:15.177124 398 log_verifier.cc:126] Checking tablet 885b10f956b54ebaa82614f8a1f979d1
I20240503 08:00:15.201809 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 885b10f956b54ebaa82614f8a1f979d1
I20240503 08:00:15.201943 398 log_verifier.cc:126] Checking tablet 8a934439594245d79ed5980ff9c03277
I20240503 08:00:15.225720 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8a934439594245d79ed5980ff9c03277
I20240503 08:00:15.225782 398 log_verifier.cc:126] Checking tablet 8efd24415e484e2a9af88b3aef6f78cc
I20240503 08:00:15.250842 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 8efd24415e484e2a9af88b3aef6f78cc
I20240503 08:00:15.250903 398 log_verifier.cc:126] Checking tablet 94f828890165466787015f21507e6ba2
I20240503 08:00:15.274562 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 94f828890165466787015f21507e6ba2
I20240503 08:00:15.274627 398 log_verifier.cc:126] Checking tablet 957ae2f17c4c41d28287d5490c54572e
I20240503 08:00:15.298394 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 957ae2f17c4c41d28287d5490c54572e
I20240503 08:00:15.298455 398 log_verifier.cc:126] Checking tablet 9a1dab998bb84cc9880252383cb1e982
I20240503 08:00:15.321995 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 9a1dab998bb84cc9880252383cb1e982
I20240503 08:00:15.322054 398 log_verifier.cc:126] Checking tablet 9e47575b08ad4f54b1f8d35db27d9e58
I20240503 08:00:15.346549 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 9e47575b08ad4f54b1f8d35db27d9e58
I20240503 08:00:15.346623 398 log_verifier.cc:126] Checking tablet a9023b7fb87a4adaa91995b883df8826
I20240503 08:00:15.373085 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet a9023b7fb87a4adaa91995b883df8826
I20240503 08:00:15.373155 398 log_verifier.cc:126] Checking tablet aa0b29181066483386cab4c6f921f8f0
I20240503 08:00:15.396600 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet aa0b29181066483386cab4c6f921f8f0
I20240503 08:00:15.396658 398 log_verifier.cc:126] Checking tablet cd0045edfdb441f2ae6920b8a97555bb
I20240503 08:00:15.420797 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet cd0045edfdb441f2ae6920b8a97555bb
I20240503 08:00:15.420861 398 log_verifier.cc:126] Checking tablet d17b7dcf55364209bd26129488dd0184
I20240503 08:00:15.444896 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet d17b7dcf55364209bd26129488dd0184
I20240503 08:00:15.444957 398 log_verifier.cc:126] Checking tablet d7012d8274214cf58c70d52062474b3a
I20240503 08:00:15.469107 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet d7012d8274214cf58c70d52062474b3a
I20240503 08:00:15.469180 398 log_verifier.cc:126] Checking tablet d74a03a296dc48e582faeddc7b978ccd
I20240503 08:00:15.497115 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet d74a03a296dc48e582faeddc7b978ccd
I20240503 08:00:15.497242 398 log_verifier.cc:126] Checking tablet d9e91e1b70504ac695eae61f72d3c149
I20240503 08:00:15.530598 398 log_verifier.cc:177] Verified matching terms for 10 ops in tablet d9e91e1b70504ac695eae61f72d3c149
I20240503 08:00:15.530671 398 log_verifier.cc:126] Checking tablet e03d7158b64f434a8871f009864a0b72
I20240503 08:00:15.561390 398 log_verifier.cc:177] Verified matching terms for 9 ops in tablet e03d7158b64f434a8871f009864a0b72
I20240503 08:00:15.561513 398 log_verifier.cc:126] Checking tablet ffd6a5145bf64148b1c9db148905f92b
I20240503 08:00:15.587499 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ffd6a5145bf64148b1c9db148905f92b
I20240503 08:00:15.587895 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 6224
I20240503 08:00:15.594766 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 6355
I20240503 08:00:15.601218 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 6486
I20240503 08:00:15.607162 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 6617
I20240503 08:00:15.612920 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 7952
I20240503 08:00:15.619518 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 8083
I20240503 08:00:15.626515 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 8214
I20240503 08:00:15.631489 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 8345
I20240503 08:00:15.636075 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 9154
I20240503 08:00:15.640604 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 6133
2024-05-03T08:00:15Z chronyd exiting
[ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (32211 ms)
[----------] 1 test from TserverAddedDuringRebalancingTest (32211 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5
I20240503 08:00:15.726398 398 test_util.cc:268] Using random seed: 472471352
I20240503 08:00:15.727121 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:00:15.727160 398 ts_itest-base.cc:113] --------------
I20240503 08:00:15.727169 398 ts_itest-base.cc:114] 4 tablet servers
I20240503 08:00:15.727178 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 08:00:15.727187 398 ts_itest-base.cc:116] --------------
2024-05-03T08:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:00:15Z Disabled control of system clock
I20240503 08:00:15.734218 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41847
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:41847
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 08:00:15.832904 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:15.833132 14215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:15.833195 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:15.836078 14215 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:15.836138 14215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:00:15.836169 14215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:15.836197 14215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:00:15.836228 14215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:00:15.839720 14215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:41847
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.14215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:15.840490 14215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:15.841517 14215 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:15.848937 14223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:15.849623 14215 server_base.cc:1034] running on GCE node
W20240503 08:00:15.850509 14220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:15.851125 14221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:15.851541 14215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:15.852336 14215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:15.853473 14215 hybrid_clock.cc:648] HybridClock initialized: now 1714723215853451 us; error 36 us; skew 500 ppm
I20240503 08:00:15.855036 14215 webserver.cc:460] Webserver started at http://127.0.99.190:35307/ using document root <none> and password file <none>
I20240503 08:00:15.855521 14215 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:15.855571 14215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:15.855723 14215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:15.857278 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "dff7fba94a8d46d7b42af59b7be96ae5"
format_stamp: "Formatted at 2024-05-03 08:00:15 on dist-test-slave-7b05"
I20240503 08:00:15.857700 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "dff7fba94a8d46d7b42af59b7be96ae5"
format_stamp: "Formatted at 2024-05-03 08:00:15 on dist-test-slave-7b05"
I20240503 08:00:15.860647 14215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:00:15.862630 14229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:15.863523 14215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:15.863619 14215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "dff7fba94a8d46d7b42af59b7be96ae5"
format_stamp: "Formatted at 2024-05-03 08:00:15 on dist-test-slave-7b05"
I20240503 08:00:15.863691 14215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:15.888587 14215 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:15.889190 14215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:15.889341 14215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:15.895196 14215 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:41847
I20240503 08:00:15.895227 14281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:41847 every 8 connection(s)
I20240503 08:00:15.896054 14215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:00:15.898470 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 14215
I20240503 08:00:15.898499 14282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:15.898591 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:00:15.904299 14282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5: Bootstrap starting.
I20240503 08:00:15.906471 14282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:15.907073 14282 log.cc:824] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:15.908470 14282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5: No bootstrap required, opened a new log
I20240503 08:00:15.910868 14282 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff7fba94a8d46d7b42af59b7be96ae5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41847 } }
I20240503 08:00:15.911029 14282 raft_consensus.cc:374] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:15.911088 14282 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff7fba94a8d46d7b42af59b7be96ae5, State: Initialized, Role: FOLLOWER
I20240503 08:00:15.911463 14282 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff7fba94a8d46d7b42af59b7be96ae5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41847 } }
I20240503 08:00:15.911602 14282 raft_consensus.cc:388] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:15.911665 14282 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:15.911762 14282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:15.912393 14282 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff7fba94a8d46d7b42af59b7be96ae5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41847 } }
I20240503 08:00:15.912671 14282 leader_election.cc:302] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff7fba94a8d46d7b42af59b7be96ae5; no voters:
I20240503 08:00:15.912887 14282 leader_election.cc:288] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:15.913229 14287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:15.913398 14287 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [term 1 LEADER]: Becoming Leader. State: Replica: dff7fba94a8d46d7b42af59b7be96ae5, State: Running, Role: LEADER
I20240503 08:00:15.913641 14287 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff7fba94a8d46d7b42af59b7be96ae5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41847 } }
I20240503 08:00:15.913826 14282 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:00:15.914959 14288 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dff7fba94a8d46d7b42af59b7be96ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff7fba94a8d46d7b42af59b7be96ae5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41847 } } }
I20240503 08:00:15.915092 14288 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:15.914968 14289 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff7fba94a8d46d7b42af59b7be96ae5. Latest consensus state: current_term: 1 leader_uuid: "dff7fba94a8d46d7b42af59b7be96ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff7fba94a8d46d7b42af59b7be96ae5" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41847 } } }
I20240503 08:00:15.915334 14289 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5 [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:15.915383 14297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:00:15.918159 14297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:00:15.922202 14297 catalog_manager.cc:1347] Generated new cluster ID: 8785acef94b54a35807d2a0c6c8488e1
I20240503 08:00:15.922273 14297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:00:15.957939 14297 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:00:15.959048 14297 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:00:15.964844 14297 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dff7fba94a8d46d7b42af59b7be96ae5: Generated new TSK 0
I20240503 08:00:15.965452 14297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:00:15.972280 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41847
--builtin_ntp_servers=127.0.99.148:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:00:16.073482 14306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:16.073696 14306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:16.073743 14306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:16.076745 14306 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:16.076812 14306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:16.076894 14306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 08:00:16.080483 14306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41847
--never_fsync=true
--heap_profile_path=/tmp/kudu.14306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:16.081429 14306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:16.082562 14306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:16.091310 14314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:16.091825 14311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:16.092804 14312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:16.093477 14306 server_base.cc:1034] running on GCE node
I20240503 08:00:16.093875 14306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:16.094432 14306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:16.095580 14306 hybrid_clock.cc:648] HybridClock initialized: now 1714723216095563 us; error 35 us; skew 500 ppm
I20240503 08:00:16.097465 14306 webserver.cc:460] Webserver started at http://127.0.99.129:40573/ using document root <none> and password file <none>
I20240503 08:00:16.098001 14306 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:16.098057 14306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:16.098219 14306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:16.099829 14306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "79cc8a5193b04055a09ca401f24842f3"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.100283 14306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "79cc8a5193b04055a09ca401f24842f3"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.103461 14306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:00:16.105787 14320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.106753 14306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:16.106853 14306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "79cc8a5193b04055a09ca401f24842f3"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.106932 14306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:16.134501 14306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:16.135103 14306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:16.135255 14306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:16.135774 14306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:16.136621 14306 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:16.136673 14306 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.136725 14306 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:16.136749 14306 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.144176 14306 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:43535
I20240503 08:00:16.144222 14433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:43535 every 8 connection(s)
I20240503 08:00:16.145045 14306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:00:16.147670 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 14306
I20240503 08:00:16.147778 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:00:16.149730 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41847
--builtin_ntp_servers=127.0.99.148:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:16.153491 14434 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41847
I20240503 08:00:16.153690 14434 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:16.154155 14434 heartbeater.cc:511] Master 127.0.99.190:41847 requested a full tablet report, sending...
I20240503 08:00:16.155462 14246 ts_manager.cc:194] Registered new tserver with Master: 79cc8a5193b04055a09ca401f24842f3 (127.0.99.129:43535)
I20240503 08:00:16.156556 14246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:37297
W20240503 08:00:16.250715 14437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:16.250910 14437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:16.250965 14437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:16.253895 14437 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:16.253957 14437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:16.254034 14437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 08:00:16.257534 14437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41847
--never_fsync=true
--heap_profile_path=/tmp/kudu.14437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:16.258347 14437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:16.259410 14437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:16.267874 14442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:16.268728 14443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:16.269418 14445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:16.270200 14437 server_base.cc:1034] running on GCE node
I20240503 08:00:16.270539 14437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:16.271054 14437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:16.272198 14437 hybrid_clock.cc:648] HybridClock initialized: now 1714723216272187 us; error 27 us; skew 500 ppm
I20240503 08:00:16.273917 14437 webserver.cc:460] Webserver started at http://127.0.99.130:36559/ using document root <none> and password file <none>
I20240503 08:00:16.274443 14437 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:16.274498 14437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:16.274672 14437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:16.276188 14437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d975dfd1169843439e8ea7b5f8df99c3"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.276621 14437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d975dfd1169843439e8ea7b5f8df99c3"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.279655 14437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:00:16.281639 14451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.282529 14437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:16.282629 14437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d975dfd1169843439e8ea7b5f8df99c3"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.282702 14437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:16.297225 14437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:16.297770 14437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:16.297909 14437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:16.298372 14437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:16.299194 14437 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:16.299239 14437 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.299288 14437 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:16.299314 14437 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.306874 14437 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39599
I20240503 08:00:16.306902 14564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39599 every 8 connection(s)
I20240503 08:00:16.307735 14437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:00:16.314625 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 14437
I20240503 08:00:16.314715 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:00:16.316293 14565 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41847
I20240503 08:00:16.316516 14565 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:16.316817 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41847
--builtin_ntp_servers=127.0.99.148:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:16.317121 14565 heartbeater.cc:511] Master 127.0.99.190:41847 requested a full tablet report, sending...
I20240503 08:00:16.318058 14246 ts_manager.cc:194] Registered new tserver with Master: d975dfd1169843439e8ea7b5f8df99c3 (127.0.99.130:39599)
I20240503 08:00:16.318778 14246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:53111
W20240503 08:00:16.418871 14568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:16.419056 14568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:16.419096 14568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:16.421990 14568 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:16.422044 14568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:16.422106 14568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:00:16.425496 14568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41847
--never_fsync=true
--heap_profile_path=/tmp/kudu.14568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:16.426210 14568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:16.427196 14568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:16.435401 14574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:16.436239 14573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:16.436853 14576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:16.437731 14568 server_base.cc:1034] running on GCE node
I20240503 08:00:16.438047 14568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:16.438522 14568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:16.439689 14568 hybrid_clock.cc:648] HybridClock initialized: now 1714723216439667 us; error 33 us; skew 500 ppm
I20240503 08:00:16.441493 14568 webserver.cc:460] Webserver started at http://127.0.99.131:41325/ using document root <none> and password file <none>
I20240503 08:00:16.441987 14568 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:16.442036 14568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:16.442199 14568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:16.443722 14568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4658a4ce8fe24ac2a9226ebe6e3e2930"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.444157 14568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4658a4ce8fe24ac2a9226ebe6e3e2930"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.447250 14568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:00:16.449257 14582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.450179 14568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:16.450291 14568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4658a4ce8fe24ac2a9226ebe6e3e2930"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.450366 14568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:16.475040 14568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:16.475617 14568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:16.475765 14568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:16.476279 14568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:16.477159 14568 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:16.477205 14568 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.477257 14568 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:16.477283 14568 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.484833 14568 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44171
I20240503 08:00:16.484856 14695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44171 every 8 connection(s)
I20240503 08:00:16.485785 14568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:00:16.492072 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 14568
I20240503 08:00:16.492190 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:00:16.493820 14696 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41847
I20240503 08:00:16.494009 14696 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:16.494256 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41847
--builtin_ntp_servers=127.0.99.148:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:16.494494 14696 heartbeater.cc:511] Master 127.0.99.190:41847 requested a full tablet report, sending...
I20240503 08:00:16.495402 14246 ts_manager.cc:194] Registered new tserver with Master: 4658a4ce8fe24ac2a9226ebe6e3e2930 (127.0.99.131:44171)
I20240503 08:00:16.495921 14246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:56799
W20240503 08:00:16.596779 14699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:16.596985 14699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:16.597041 14699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:16.600015 14699 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:16.600078 14699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:16.600162 14699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:00:16.603621 14699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41847
--never_fsync=true
--heap_profile_path=/tmp/kudu.14699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:16.604426 14699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:16.605485 14699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:16.613584 14707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:16.614464 14704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:16.615165 14705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:16.615911 14699 server_base.cc:1034] running on GCE node
I20240503 08:00:16.616256 14699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:16.616775 14699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:16.617916 14699 hybrid_clock.cc:648] HybridClock initialized: now 1714723216617892 us; error 34 us; skew 500 ppm
I20240503 08:00:16.619644 14699 webserver.cc:460] Webserver started at http://127.0.99.132:37531/ using document root <none> and password file <none>
I20240503 08:00:16.620142 14699 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:16.620193 14699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:16.620363 14699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:16.621966 14699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fc08de44875242358dc0eab5e7ebd5a0"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.622392 14699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fc08de44875242358dc0eab5e7ebd5a0"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.625387 14699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:00:16.627295 14713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.628201 14699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:16.628289 14699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fc08de44875242358dc0eab5e7ebd5a0"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:16.628372 14699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:16.643304 14699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:16.643934 14699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:16.644089 14699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:16.644563 14699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:16.645610 14699 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:16.645666 14699 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.645713 14699 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:16.645740 14699 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:16.653463 14699 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:39343
I20240503 08:00:16.653481 14826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:39343 every 8 connection(s)
I20240503 08:00:16.654307 14699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:00:16.659786 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 14699
I20240503 08:00:16.659880 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:00:16.662226 14827 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41847
I20240503 08:00:16.662420 14827 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:16.662870 14827 heartbeater.cc:511] Master 127.0.99.190:41847 requested a full tablet report, sending...
I20240503 08:00:16.663686 14246 ts_manager.cc:194] Registered new tserver with Master: fc08de44875242358dc0eab5e7ebd5a0 (127.0.99.132:39343)
I20240503 08:00:16.664175 14246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:39039
I20240503 08:00:16.673280 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:00:16.681028 14246 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45966:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:00:16.695394 14499 tablet_service.cc:1467] Processing CreateTablet for tablet 5a59ff7ed90c40409dc3fea316255cb3 (DEFAULT_TABLE table=TestTable [id=675bdd0183ca4a79acfc185ce7148abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:16.696204 14499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a59ff7ed90c40409dc3fea316255cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:16.700755 14840 tablet_bootstrap.cc:492] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3: Bootstrap starting.
I20240503 08:00:16.702518 14840 tablet_bootstrap.cc:654] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:16.703114 14840 log.cc:824] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:16.704444 14840 tablet_bootstrap.cc:492] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3: No bootstrap required, opened a new log
I20240503 08:00:16.704591 14840 ts_tablet_manager.cc:1367] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:00:16.707360 14840 raft_consensus.cc:348] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:16.707516 14840 raft_consensus.cc:374] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:16.707561 14840 raft_consensus.cc:729] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Initialized, Role: FOLLOWER
I20240503 08:00:16.707898 14840 consensus_queue.cc:260] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:16.708004 14840 raft_consensus.cc:388] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:16.708060 14840 raft_consensus.cc:482] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:16.708158 14840 raft_consensus.cc:3037] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:16.708814 14840 raft_consensus.cc:504] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:16.709113 14840 leader_election.cc:302] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d975dfd1169843439e8ea7b5f8df99c3; no voters:
I20240503 08:00:16.709362 14840 leader_election.cc:288] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:16.709441 14842 raft_consensus.cc:2781] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:16.709632 14842 raft_consensus.cc:686] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 LEADER]: Becoming Leader. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Running, Role: LEADER
I20240503 08:00:16.709900 14842 consensus_queue.cc:237] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:16.710000 14565 heartbeater.cc:503] Master 127.0.99.190:41847 was elected leader, sending a full tablet report...
I20240503 08:00:16.710105 14840 ts_tablet_manager.cc:1398] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 08:00:16.712806 14246 catalog_manager.cc:5484] T 5a59ff7ed90c40409dc3fea316255cb3 P d975dfd1169843439e8ea7b5f8df99c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d975dfd1169843439e8ea7b5f8df99c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d975dfd1169843439e8ea7b5f8df99c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:16.720424 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:00:16.721547 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 79cc8a5193b04055a09ca401f24842f3 to finish bootstrapping
I20240503 08:00:16.726729 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d975dfd1169843439e8ea7b5f8df99c3 to finish bootstrapping
I20240503 08:00:16.731037 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4658a4ce8fe24ac2a9226ebe6e3e2930 to finish bootstrapping
I20240503 08:00:16.735997 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fc08de44875242358dc0eab5e7ebd5a0 to finish bootstrapping
I20240503 08:00:16.740748 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 14568
I20240503 08:00:16.747251 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 14699
I20240503 08:00:17.158663 14434 heartbeater.cc:503] Master 127.0.99.190:41847 was elected leader, sending a full tablet report...
I20240503 08:00:20.502916 14246 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45966:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:20.513424 14367 tablet_service.cc:1467] Processing CreateTablet for tablet 54fdf23525bb412e8e45faa2efdc745a (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a2dabdb0c3a4cb4b2a10b8f4db26a6c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:20.513437 14368 tablet_service.cc:1467] Processing CreateTablet for tablet 6e35adbe05654a789b7c3ba9bf068709 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a2dabdb0c3a4cb4b2a10b8f4db26a6c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:20.513437 14366 tablet_service.cc:1467] Processing CreateTablet for tablet 656a393344e14c0ab46f6ec995d964b5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a2dabdb0c3a4cb4b2a10b8f4db26a6c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:20.514457 14368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e35adbe05654a789b7c3ba9bf068709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.514798 14367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fdf23525bb412e8e45faa2efdc745a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.514964 14366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 656a393344e14c0ab46f6ec995d964b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.519610 14857 tablet_bootstrap.cc:492] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3: Bootstrap starting.
I20240503 08:00:20.521212 14857 tablet_bootstrap.cc:654] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:20.522112 14857 log.cc:824] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:20.523419 14857 tablet_bootstrap.cc:492] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3: No bootstrap required, opened a new log
I20240503 08:00:20.523566 14857 ts_tablet_manager.cc:1367] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:00:20.525794 14857 raft_consensus.cc:348] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.525976 14857 raft_consensus.cc:374] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:20.526046 14857 raft_consensus.cc:729] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Initialized, Role: FOLLOWER
I20240503 08:00:20.526504 14857 consensus_queue.cc:260] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.526618 14857 raft_consensus.cc:388] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:20.526659 14857 raft_consensus.cc:482] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:20.526729 14857 raft_consensus.cc:3037] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:20.527381 14857 raft_consensus.cc:504] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.527647 14857 leader_election.cc:302] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cc8a5193b04055a09ca401f24842f3; no voters:
I20240503 08:00:20.527856 14857 leader_election.cc:288] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:20.527941 14859 raft_consensus.cc:2781] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:20.528136 14859 raft_consensus.cc:686] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [term 1 LEADER]: Becoming Leader. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Running, Role: LEADER
I20240503 08:00:20.528431 14859 consensus_queue.cc:237] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.528558 14857 ts_tablet_manager.cc:1398] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:00:20.528754 14857 tablet_bootstrap.cc:492] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3: Bootstrap starting.
I20240503 08:00:20.529508 14857 tablet_bootstrap.cc:654] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:20.530464 14857 tablet_bootstrap.cc:492] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3: No bootstrap required, opened a new log
I20240503 08:00:20.530580 14857 ts_tablet_manager.cc:1367] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:20.530658 14246 catalog_manager.cc:5484] T 54fdf23525bb412e8e45faa2efdc745a P 79cc8a5193b04055a09ca401f24842f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79cc8a5193b04055a09ca401f24842f3 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "79cc8a5193b04055a09ca401f24842f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:20.531037 14857 raft_consensus.cc:348] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.531114 14857 raft_consensus.cc:374] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:20.531144 14857 raft_consensus.cc:729] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Initialized, Role: FOLLOWER
I20240503 08:00:20.531240 14857 consensus_queue.cc:260] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.531314 14857 raft_consensus.cc:388] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:20.531380 14857 raft_consensus.cc:482] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:20.531430 14857 raft_consensus.cc:3037] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:20.532119 14857 raft_consensus.cc:504] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.532212 14857 leader_election.cc:302] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cc8a5193b04055a09ca401f24842f3; no voters:
I20240503 08:00:20.532291 14857 leader_election.cc:288] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:20.532339 14859 raft_consensus.cc:2781] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:20.532459 14859 raft_consensus.cc:686] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [term 1 LEADER]: Becoming Leader. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Running, Role: LEADER
I20240503 08:00:20.532567 14859 consensus_queue.cc:237] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.532637 14857 ts_tablet_manager.cc:1398] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:20.532747 14857 tablet_bootstrap.cc:492] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3: Bootstrap starting.
I20240503 08:00:20.533541 14857 tablet_bootstrap.cc:654] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:20.534037 14246 catalog_manager.cc:5484] T 6e35adbe05654a789b7c3ba9bf068709 P 79cc8a5193b04055a09ca401f24842f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79cc8a5193b04055a09ca401f24842f3 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "79cc8a5193b04055a09ca401f24842f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:20.534533 14857 tablet_bootstrap.cc:492] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3: No bootstrap required, opened a new log
I20240503 08:00:20.534593 14857 ts_tablet_manager.cc:1367] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:20.534986 14857 raft_consensus.cc:348] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.535097 14857 raft_consensus.cc:374] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:20.535141 14857 raft_consensus.cc:729] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Initialized, Role: FOLLOWER
I20240503 08:00:20.535243 14857 consensus_queue.cc:260] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.535313 14857 raft_consensus.cc:388] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:20.535336 14857 raft_consensus.cc:482] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:20.535375 14857 raft_consensus.cc:3037] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:20.536032 14857 raft_consensus.cc:504] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.536125 14857 leader_election.cc:302] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cc8a5193b04055a09ca401f24842f3; no voters:
I20240503 08:00:20.536206 14857 leader_election.cc:288] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:20.536257 14861 raft_consensus.cc:2781] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:20.536358 14861 raft_consensus.cc:686] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [term 1 LEADER]: Becoming Leader. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Running, Role: LEADER
I20240503 08:00:20.536402 14857 ts_tablet_manager.cc:1398] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:20.536471 14861 consensus_queue.cc:237] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.537580 14246 catalog_manager.cc:5484] T 656a393344e14c0ab46f6ec995d964b5 P 79cc8a5193b04055a09ca401f24842f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79cc8a5193b04055a09ca401f24842f3 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "79cc8a5193b04055a09ca401f24842f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:20.703644 14246 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45966:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:20.708709 14499 tablet_service.cc:1467] Processing CreateTablet for tablet c90faaf0e9f948ee9117005b7ccd9d2b (DEFAULT_TABLE table=rebalance_test_table_1 [id=edea72be6d1144d3879cfc23bf989dd4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:20.708712 14498 tablet_service.cc:1467] Processing CreateTablet for tablet 133dfc52591b4d1aba37fbc1ff1ba622 (DEFAULT_TABLE table=rebalance_test_table_1 [id=edea72be6d1144d3879cfc23bf989dd4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:20.708717 14497 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8e7845420b474f9fefe788b81a8eb1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=edea72be6d1144d3879cfc23bf989dd4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:20.709131 14499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c90faaf0e9f948ee9117005b7ccd9d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.709318 14497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e8e7845420b474f9fefe788b81a8eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.710427 14498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133dfc52591b4d1aba37fbc1ff1ba622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.711570 14887 tablet_bootstrap.cc:492] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3: Bootstrap starting.
I20240503 08:00:20.712415 14887 tablet_bootstrap.cc:654] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:20.713404 14887 tablet_bootstrap.cc:492] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3: No bootstrap required, opened a new log
I20240503 08:00:20.713482 14887 ts_tablet_manager.cc:1367] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:20.713999 14887 raft_consensus.cc:348] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.714108 14887 raft_consensus.cc:374] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:20.714154 14887 raft_consensus.cc:729] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Initialized, Role: FOLLOWER
I20240503 08:00:20.714291 14887 consensus_queue.cc:260] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.714388 14887 raft_consensus.cc:388] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:20.714435 14887 raft_consensus.cc:482] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:20.714488 14887 raft_consensus.cc:3037] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:20.715231 14887 raft_consensus.cc:504] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.715354 14887 leader_election.cc:302] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d975dfd1169843439e8ea7b5f8df99c3; no voters:
I20240503 08:00:20.715508 14887 leader_election.cc:288] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:20.715591 14889 raft_consensus.cc:2781] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:20.715796 14887 ts_tablet_manager.cc:1398] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:20.715852 14889 raft_consensus.cc:686] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [term 1 LEADER]: Becoming Leader. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Running, Role: LEADER
I20240503 08:00:20.715965 14887 tablet_bootstrap.cc:492] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3: Bootstrap starting.
I20240503 08:00:20.715987 14889 consensus_queue.cc:237] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.716826 14887 tablet_bootstrap.cc:654] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:20.717346 14246 catalog_manager.cc:5484] T c90faaf0e9f948ee9117005b7ccd9d2b P d975dfd1169843439e8ea7b5f8df99c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d975dfd1169843439e8ea7b5f8df99c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d975dfd1169843439e8ea7b5f8df99c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:20.717929 14887 tablet_bootstrap.cc:492] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3: No bootstrap required, opened a new log
I20240503 08:00:20.718003 14887 ts_tablet_manager.cc:1367] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:20.718329 14887 raft_consensus.cc:348] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.718398 14887 raft_consensus.cc:374] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:20.718417 14887 raft_consensus.cc:729] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Initialized, Role: FOLLOWER
I20240503 08:00:20.718506 14887 consensus_queue.cc:260] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.718566 14887 raft_consensus.cc:388] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:20.718595 14887 raft_consensus.cc:482] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:20.718639 14887 raft_consensus.cc:3037] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:20.719590 14887 raft_consensus.cc:504] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.719694 14887 leader_election.cc:302] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d975dfd1169843439e8ea7b5f8df99c3; no voters:
I20240503 08:00:20.719769 14887 leader_election.cc:288] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:20.719805 14889 raft_consensus.cc:2781] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:20.719884 14889 raft_consensus.cc:686] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 LEADER]: Becoming Leader. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Running, Role: LEADER
I20240503 08:00:20.719997 14887 ts_tablet_manager.cc:1398] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:20.719980 14889 consensus_queue.cc:237] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.720131 14887 tablet_bootstrap.cc:492] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3: Bootstrap starting.
I20240503 08:00:20.720880 14887 tablet_bootstrap.cc:654] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:20.721815 14246 catalog_manager.cc:5484] T 2e8e7845420b474f9fefe788b81a8eb1 P d975dfd1169843439e8ea7b5f8df99c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d975dfd1169843439e8ea7b5f8df99c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d975dfd1169843439e8ea7b5f8df99c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:20.722661 14887 tablet_bootstrap.cc:492] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3: No bootstrap required, opened a new log
I20240503 08:00:20.722728 14887 ts_tablet_manager.cc:1367] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:20.723091 14887 raft_consensus.cc:348] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.723176 14887 raft_consensus.cc:374] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:20.723206 14887 raft_consensus.cc:729] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Initialized, Role: FOLLOWER
I20240503 08:00:20.723304 14887 consensus_queue.cc:260] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.723361 14887 raft_consensus.cc:388] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:20.723382 14887 raft_consensus.cc:482] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:20.723407 14887 raft_consensus.cc:3037] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:20.723973 14887 raft_consensus.cc:504] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.724073 14887 leader_election.cc:302] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d975dfd1169843439e8ea7b5f8df99c3; no voters:
I20240503 08:00:20.724150 14887 leader_election.cc:288] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:20.724196 14889 raft_consensus.cc:2781] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:20.724267 14889 raft_consensus.cc:686] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 LEADER]: Becoming Leader. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Running, Role: LEADER
I20240503 08:00:20.724345 14887 ts_tablet_manager.cc:1398] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:00:20.724363 14889 consensus_queue.cc:237] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:20.725440 14246 catalog_manager.cc:5484] T 133dfc52591b4d1aba37fbc1ff1ba622 P d975dfd1169843439e8ea7b5f8df99c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d975dfd1169843439e8ea7b5f8df99c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d975dfd1169843439e8ea7b5f8df99c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:20.877998 14246 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45966:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:20.882786 14366 tablet_service.cc:1467] Processing CreateTablet for tablet 720ea37227a04513bbc9eed34966a411 (DEFAULT_TABLE table=rebalance_test_table_2 [id=965e0e55a25d426d9515d199e54127ac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:20.882786 14368 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4fe5222528486580b2c154619f0c5a (DEFAULT_TABLE table=rebalance_test_table_2 [id=965e0e55a25d426d9515d199e54127ac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:20.882843 14367 tablet_service.cc:1467] Processing CreateTablet for tablet ad05f41e2f4348f2b60c3bc335851fcf (DEFAULT_TABLE table=rebalance_test_table_2 [id=965e0e55a25d426d9515d199e54127ac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:20.883126 14368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4fe5222528486580b2c154619f0c5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.883278 14367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad05f41e2f4348f2b60c3bc335851fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.883404 14366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720ea37227a04513bbc9eed34966a411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:20.884812 14857 tablet_bootstrap.cc:492] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3: Bootstrap starting.
I20240503 08:00:20.885699 14857 tablet_bootstrap.cc:654] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:20.886750 14857 tablet_bootstrap.cc:492] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3: No bootstrap required, opened a new log
I20240503 08:00:20.886816 14857 ts_tablet_manager.cc:1367] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:20.887162 14857 raft_consensus.cc:348] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.887240 14857 raft_consensus.cc:374] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:20.887266 14857 raft_consensus.cc:729] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Initialized, Role: FOLLOWER
I20240503 08:00:20.887353 14857 consensus_queue.cc:260] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.887404 14857 raft_consensus.cc:388] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:20.887429 14857 raft_consensus.cc:482] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:20.887459 14857 raft_consensus.cc:3037] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:20.888021 14857 raft_consensus.cc:504] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.888134 14857 leader_election.cc:302] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cc8a5193b04055a09ca401f24842f3; no voters:
I20240503 08:00:20.888263 14861 raft_consensus.cc:2781] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:20.888339 14861 raft_consensus.cc:686] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [term 1 LEADER]: Becoming Leader. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Running, Role: LEADER
I20240503 08:00:20.888455 14861 consensus_queue.cc:237] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.888805 14857 leader_election.cc:288] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:20.889024 14857 ts_tablet_manager.cc:1398] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:20.889192 14857 tablet_bootstrap.cc:492] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3: Bootstrap starting.
I20240503 08:00:20.889595 14246 catalog_manager.cc:5484] T ad05f41e2f4348f2b60c3bc335851fcf P 79cc8a5193b04055a09ca401f24842f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79cc8a5193b04055a09ca401f24842f3 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "79cc8a5193b04055a09ca401f24842f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:20.889890 14857 tablet_bootstrap.cc:654] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:20.890718 14857 tablet_bootstrap.cc:492] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3: No bootstrap required, opened a new log
I20240503 08:00:20.890796 14857 ts_tablet_manager.cc:1367] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:20.891196 14857 raft_consensus.cc:348] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.891278 14857 raft_consensus.cc:374] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:20.891305 14857 raft_consensus.cc:729] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Initialized, Role: FOLLOWER
I20240503 08:00:20.891409 14857 consensus_queue.cc:260] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.891480 14857 raft_consensus.cc:388] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:20.891507 14857 raft_consensus.cc:482] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:20.891542 14857 raft_consensus.cc:3037] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:20.892110 14857 raft_consensus.cc:504] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.892211 14857 leader_election.cc:302] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cc8a5193b04055a09ca401f24842f3; no voters:
I20240503 08:00:20.892305 14857 leader_election.cc:288] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:20.892349 14861 raft_consensus.cc:2781] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:20.892429 14861 raft_consensus.cc:686] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [term 1 LEADER]: Becoming Leader. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Running, Role: LEADER
I20240503 08:00:20.892526 14861 consensus_queue.cc:237] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.892586 14857 ts_tablet_manager.cc:1398] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:20.892729 14857 tablet_bootstrap.cc:492] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3: Bootstrap starting.
I20240503 08:00:20.893555 14857 tablet_bootstrap.cc:654] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:20.894441 14857 tablet_bootstrap.cc:492] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3: No bootstrap required, opened a new log
I20240503 08:00:20.894505 14857 ts_tablet_manager.cc:1367] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:20.894796 14245 catalog_manager.cc:5484] T 720ea37227a04513bbc9eed34966a411 P 79cc8a5193b04055a09ca401f24842f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79cc8a5193b04055a09ca401f24842f3 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "79cc8a5193b04055a09ca401f24842f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:20.894851 14857 raft_consensus.cc:348] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.894927 14857 raft_consensus.cc:374] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:20.894965 14857 raft_consensus.cc:729] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Initialized, Role: FOLLOWER
I20240503 08:00:20.895064 14857 consensus_queue.cc:260] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.895153 14857 raft_consensus.cc:388] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:20.895176 14857 raft_consensus.cc:482] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:20.895213 14857 raft_consensus.cc:3037] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:20.895879 14857 raft_consensus.cc:504] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.895978 14857 leader_election.cc:302] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cc8a5193b04055a09ca401f24842f3; no voters:
I20240503 08:00:20.896057 14857 leader_election.cc:288] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:20.896113 14861 raft_consensus.cc:2781] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:20.896193 14861 raft_consensus.cc:686] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [term 1 LEADER]: Becoming Leader. State: Replica: 79cc8a5193b04055a09ca401f24842f3, State: Running, Role: LEADER
I20240503 08:00:20.896279 14861 consensus_queue.cc:237] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } }
I20240503 08:00:20.896317 14857 ts_tablet_manager.cc:1398] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:20.897723 14245 catalog_manager.cc:5484] T 2f4fe5222528486580b2c154619f0c5a P 79cc8a5193b04055a09ca401f24842f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79cc8a5193b04055a09ca401f24842f3 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "79cc8a5193b04055a09ca401f24842f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc8a5193b04055a09ca401f24842f3" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43535 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:21.045967 14245 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45966:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:00:21.050570 14498 tablet_service.cc:1467] Processing CreateTablet for tablet 4056887efea94564b7dd4eeb34bb4e6f (DEFAULT_TABLE table=rebalance_test_table_3 [id=97216865e25b4d56b88d17468464fec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:21.050650 14497 tablet_service.cc:1467] Processing CreateTablet for tablet fc826379f51043499fdf053d1ab0e445 (DEFAULT_TABLE table=rebalance_test_table_3 [id=97216865e25b4d56b88d17468464fec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:21.050643 14499 tablet_service.cc:1467] Processing CreateTablet for tablet 0437a21d93c5408ba4bf9b8f84c0d7da (DEFAULT_TABLE table=rebalance_test_table_3 [id=97216865e25b4d56b88d17468464fec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:21.050861 14498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4056887efea94564b7dd4eeb34bb4e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.050971 14497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc826379f51043499fdf053d1ab0e445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.051074 14499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0437a21d93c5408ba4bf9b8f84c0d7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:21.052541 14887 tablet_bootstrap.cc:492] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3: Bootstrap starting.
I20240503 08:00:21.053369 14887 tablet_bootstrap.cc:654] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:21.054239 14887 tablet_bootstrap.cc:492] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3: No bootstrap required, opened a new log
I20240503 08:00:21.054306 14887 ts_tablet_manager.cc:1367] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.054641 14887 raft_consensus.cc:348] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.054719 14887 raft_consensus.cc:374] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:21.054746 14887 raft_consensus.cc:729] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Initialized, Role: FOLLOWER
I20240503 08:00:21.054852 14887 consensus_queue.cc:260] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.054920 14887 raft_consensus.cc:388] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:21.054955 14887 raft_consensus.cc:482] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:21.054998 14887 raft_consensus.cc:3037] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:21.055584 14887 raft_consensus.cc:504] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.055693 14887 leader_election.cc:302] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d975dfd1169843439e8ea7b5f8df99c3; no voters:
I20240503 08:00:21.055775 14887 leader_election.cc:288] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:21.055821 14889 raft_consensus.cc:2781] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:21.055929 14889 raft_consensus.cc:686] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [term 1 LEADER]: Becoming Leader. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Running, Role: LEADER
I20240503 08:00:21.056034 14889 consensus_queue.cc:237] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.056249 14887 ts_tablet_manager.cc:1398] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:21.056389 14887 tablet_bootstrap.cc:492] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3: Bootstrap starting.
I20240503 08:00:21.057329 14887 tablet_bootstrap.cc:654] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:21.057601 14245 catalog_manager.cc:5484] T fc826379f51043499fdf053d1ab0e445 P d975dfd1169843439e8ea7b5f8df99c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d975dfd1169843439e8ea7b5f8df99c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d975dfd1169843439e8ea7b5f8df99c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:21.058190 14887 tablet_bootstrap.cc:492] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3: No bootstrap required, opened a new log
I20240503 08:00:21.058259 14887 ts_tablet_manager.cc:1367] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:21.058645 14887 raft_consensus.cc:348] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.058727 14887 raft_consensus.cc:374] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:21.058758 14887 raft_consensus.cc:729] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Initialized, Role: FOLLOWER
I20240503 08:00:21.058869 14887 consensus_queue.cc:260] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.058943 14887 raft_consensus.cc:388] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:21.058977 14887 raft_consensus.cc:482] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:21.059011 14887 raft_consensus.cc:3037] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:21.059659 14887 raft_consensus.cc:504] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.059762 14887 leader_election.cc:302] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d975dfd1169843439e8ea7b5f8df99c3; no voters:
I20240503 08:00:21.059857 14887 leader_election.cc:288] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:21.059917 14889 raft_consensus.cc:2781] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:21.060057 14887 ts_tablet_manager.cc:1398] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:00:21.060129 14889 raft_consensus.cc:686] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [term 1 LEADER]: Becoming Leader. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Running, Role: LEADER
I20240503 08:00:21.060187 14887 tablet_bootstrap.cc:492] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3: Bootstrap starting.
I20240503 08:00:21.060241 14889 consensus_queue.cc:237] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.060942 14887 tablet_bootstrap.cc:654] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:21.061395 14245 catalog_manager.cc:5484] T 4056887efea94564b7dd4eeb34bb4e6f P d975dfd1169843439e8ea7b5f8df99c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d975dfd1169843439e8ea7b5f8df99c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d975dfd1169843439e8ea7b5f8df99c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:21.061843 14887 tablet_bootstrap.cc:492] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3: No bootstrap required, opened a new log
I20240503 08:00:21.061913 14887 ts_tablet_manager.cc:1367] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:21.062225 14887 raft_consensus.cc:348] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.062295 14887 raft_consensus.cc:374] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:21.062323 14887 raft_consensus.cc:729] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Initialized, Role: FOLLOWER
I20240503 08:00:21.062427 14887 consensus_queue.cc:260] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.062507 14887 raft_consensus.cc:388] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:21.062544 14887 raft_consensus.cc:482] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:21.062587 14887 raft_consensus.cc:3037] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:21.063179 14887 raft_consensus.cc:504] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.063288 14887 leader_election.cc:302] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d975dfd1169843439e8ea7b5f8df99c3; no voters:
I20240503 08:00:21.063372 14887 leader_election.cc:288] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:21.063426 14890 raft_consensus.cc:2781] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:21.063515 14890 raft_consensus.cc:686] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [term 1 LEADER]: Becoming Leader. State: Replica: d975dfd1169843439e8ea7b5f8df99c3, State: Running, Role: LEADER
I20240503 08:00:21.063566 14887 ts_tablet_manager.cc:1398] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:00:21.063606 14890 consensus_queue.cc:237] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } }
I20240503 08:00:21.066123 14246 catalog_manager.cc:5484] T 0437a21d93c5408ba4bf9b8f84c0d7da P d975dfd1169843439e8ea7b5f8df99c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d975dfd1169843439e8ea7b5f8df99c3 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d975dfd1169843439e8ea7b5f8df99c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975dfd1169843439e8ea7b5f8df99c3" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39599 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:21.211309 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:44171
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:41847
--webserver_port=41325
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:00:21.309501 14963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:21.309712 14963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:21.309762 14963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:21.312705 14963 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:21.312764 14963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:21.312841 14963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:00:21.316445 14963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:44171
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=41325
--tserver_master_addrs=127.0.99.190:41847
--never_fsync=true
--heap_profile_path=/tmp/kudu.14963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:21.317277 14963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:21.318317 14963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:21.326325 14968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:21.327215 14969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:21.327836 14971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:21.328691 14963 server_base.cc:1034] running on GCE node
I20240503 08:00:21.329017 14963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:21.329528 14963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:21.330667 14963 hybrid_clock.cc:648] HybridClock initialized: now 1714723221330656 us; error 27 us; skew 500 ppm
I20240503 08:00:21.332304 14963 webserver.cc:460] Webserver started at http://127.0.99.131:41325/ using document root <none> and password file <none>
I20240503 08:00:21.332791 14963 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:21.332859 14963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:21.335877 14963 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:21.337558 14977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:21.338469 14963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:21.338560 14963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4658a4ce8fe24ac2a9226ebe6e3e2930"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:21.338912 14963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:21.367980 14963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:21.368541 14963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:21.368686 14963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:21.369199 14963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:21.370014 14963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:21.370060 14963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:21.370118 14963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:21.370144 14963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:21.377897 14963 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44171
I20240503 08:00:21.377946 15090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44171 every 8 connection(s)
I20240503 08:00:21.378852 14963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:00:21.386088 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 14963
I20240503 08:00:21.386643 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:39343
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:41847
--webserver_port=37531
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:21.386972 15091 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41847
I20240503 08:00:21.387171 15091 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:21.387631 15091 heartbeater.cc:511] Master 127.0.99.190:41847 requested a full tablet report, sending...
I20240503 08:00:21.388643 14246 ts_manager.cc:194] Re-registered known tserver with Master: 4658a4ce8fe24ac2a9226ebe6e3e2930 (127.0.99.131:44171)
I20240503 08:00:21.389205 14246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:51453
W20240503 08:00:21.487017 15094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:21.487288 15094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:21.487380 15094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:21.490372 15094 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:21.490432 15094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:21.490509 15094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:00:21.493980 15094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:39343
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=37531
--tserver_master_addrs=127.0.99.190:41847
--never_fsync=true
--heap_profile_path=/tmp/kudu.15094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:21.494788 15094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:21.495805 15094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:21.504082 15102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:21.504880 15099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:21.505475 15100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:21.506232 15094 server_base.cc:1034] running on GCE node
I20240503 08:00:21.506582 15094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:21.507090 15094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:21.508260 15094 hybrid_clock.cc:648] HybridClock initialized: now 1714723221508238 us; error 36 us; skew 500 ppm
I20240503 08:00:21.509987 15094 webserver.cc:460] Webserver started at http://127.0.99.132:37531/ using document root <none> and password file <none>
I20240503 08:00:21.510480 15094 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:21.510547 15094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:21.513545 15094 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:00:21.515256 15108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:21.516162 15094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:21.516253 15094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fc08de44875242358dc0eab5e7ebd5a0"
format_stamp: "Formatted at 2024-05-03 08:00:16 on dist-test-slave-7b05"
I20240503 08:00:21.516633 15094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:21.524894 15094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:21.525364 15094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:21.525489 15094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:21.525899 15094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:21.526671 15094 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:21.526713 15094 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:21.526751 15094 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:21.526777 15094 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:21.533941 15094 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:39343
I20240503 08:00:21.533975 15221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:39343 every 8 connection(s)
I20240503 08:00:21.534817 15094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:00:21.541647 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 15094
I20240503 08:00:21.542829 15222 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41847
I20240503 08:00:21.543027 15222 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:21.543517 15222 heartbeater.cc:511] Master 127.0.99.190:41847 requested a full tablet report, sending...
I20240503 08:00:21.544423 14246 ts_manager.cc:194] Re-registered known tserver with Master: fc08de44875242358dc0eab5e7ebd5a0 (127.0.99.132:39343)
I20240503 08:00:21.545096 14246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:58663
I20240503 08:00:21.712314 14368 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:21.712913 14499 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.715194 15025 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.715965 15156 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.810810 15156 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.811148 14368 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:21.811754 14499 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.811834 15025 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.895567 15025 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:21.896947 14368 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:21.903748 14499 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:00:21.917646 15156 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
dff7fba94a8d46d7b42af59b7be96ae5 | 127.0.99.190:41847 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:40707 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4658a4ce8fe24ac2a9226ebe6e3e2930 | 127.0.99.131:44171 | HEALTHY | <none> | 0 | 0
79cc8a5193b04055a09ca401f24842f3 | 127.0.99.129:43535 | HEALTHY | <none> | 6 | 0
d975dfd1169843439e8ea7b5f8df99c3 | 127.0.99.130:39599 | HEALTHY | <none> | 7 | 0
fc08de44875242358dc0eab5e7ebd5a0 | 127.0.99.132:39343 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43535
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:39599
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:44171
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:39343
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43535
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:39599
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:44171
server_dump_info_path | /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:39343
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:40707 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:00:21.935155 398 log_verifier.cc:126] Checking tablet 0437a21d93c5408ba4bf9b8f84c0d7da
I20240503 08:00:21.942584 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0437a21d93c5408ba4bf9b8f84c0d7da
I20240503 08:00:21.942644 398 log_verifier.cc:126] Checking tablet 133dfc52591b4d1aba37fbc1ff1ba622
I20240503 08:00:21.949345 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 133dfc52591b4d1aba37fbc1ff1ba622
I20240503 08:00:21.949400 398 log_verifier.cc:126] Checking tablet 2e8e7845420b474f9fefe788b81a8eb1
I20240503 08:00:21.956000 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2e8e7845420b474f9fefe788b81a8eb1
I20240503 08:00:21.956063 398 log_verifier.cc:126] Checking tablet 2f4fe5222528486580b2c154619f0c5a
I20240503 08:00:21.962836 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 2f4fe5222528486580b2c154619f0c5a
I20240503 08:00:21.962893 398 log_verifier.cc:126] Checking tablet 4056887efea94564b7dd4eeb34bb4e6f
I20240503 08:00:21.969651 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 4056887efea94564b7dd4eeb34bb4e6f
I20240503 08:00:21.969705 398 log_verifier.cc:126] Checking tablet 54fdf23525bb412e8e45faa2efdc745a
I20240503 08:00:21.976332 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 54fdf23525bb412e8e45faa2efdc745a
I20240503 08:00:21.976383 398 log_verifier.cc:126] Checking tablet 5a59ff7ed90c40409dc3fea316255cb3
I20240503 08:00:21.983031 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5a59ff7ed90c40409dc3fea316255cb3
I20240503 08:00:21.983080 398 log_verifier.cc:126] Checking tablet 656a393344e14c0ab46f6ec995d964b5
I20240503 08:00:21.989722 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 656a393344e14c0ab46f6ec995d964b5
I20240503 08:00:21.989771 398 log_verifier.cc:126] Checking tablet 6e35adbe05654a789b7c3ba9bf068709
I20240503 08:00:21.996439 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 6e35adbe05654a789b7c3ba9bf068709
I20240503 08:00:21.996492 398 log_verifier.cc:126] Checking tablet 720ea37227a04513bbc9eed34966a411
I20240503 08:00:22.003031 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 720ea37227a04513bbc9eed34966a411
I20240503 08:00:22.003080 398 log_verifier.cc:126] Checking tablet ad05f41e2f4348f2b60c3bc335851fcf
I20240503 08:00:22.009598 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet ad05f41e2f4348f2b60c3bc335851fcf
I20240503 08:00:22.009646 398 log_verifier.cc:126] Checking tablet c90faaf0e9f948ee9117005b7ccd9d2b
I20240503 08:00:22.016530 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet c90faaf0e9f948ee9117005b7ccd9d2b
I20240503 08:00:22.016584 398 log_verifier.cc:126] Checking tablet fc826379f51043499fdf053d1ab0e445
I20240503 08:00:22.023295 398 log_verifier.cc:177] Verified matching terms for 1 ops in tablet fc826379f51043499fdf053d1ab0e445
I20240503 08:00:22.024067 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 14306
I20240503 08:00:22.029255 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 14437
I20240503 08:00:22.033854 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 14963
I20240503 08:00:22.038280 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 15094
I20240503 08:00:22.042615 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 14215
2024-05-03T08:00:22Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (6345 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (6345 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1
I20240503 08:00:22.071751 398 test_util.cc:268] Using random seed: 478816706
I20240503 08:00:22.072479 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:00:22.072515 398 ts_itest-base.cc:113] --------------
I20240503 08:00:22.072525 398 ts_itest-base.cc:114] 4 tablet servers
I20240503 08:00:22.072535 398 ts_itest-base.cc:115] 2 replicas per TS
I20240503 08:00:22.072556 398 ts_itest-base.cc:116] --------------
2024-05-03T08:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:00:22Z Disabled control of system clock
I20240503 08:00:22.079926 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34401
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:41421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:34401
--location_mapping_cmd=/tmp/dist-test-taskD2FJ4a/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 08:00:22.176471 15330 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:00:22.176678 15330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:22.176703 15330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:22.176743 15330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:22.179682 15330 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:22.179746 15330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:00:22.179770 15330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:22.179792 15330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:00:22.179807 15330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:00:22.183291 15330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskD2FJ4a/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:34401
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.15330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:22.184006 15330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:22.184950 15330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:22.192140 15338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.192947 15335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.193912 15336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:22.194483 15330 server_base.cc:1034] running on GCE node
I20240503 08:00:22.194940 15330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:22.195755 15330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:22.196914 15330 hybrid_clock.cc:648] HybridClock initialized: now 1714723222196888 us; error 36 us; skew 500 ppm
I20240503 08:00:22.198614 15330 webserver.cc:460] Webserver started at http://127.0.99.190:42653/ using document root <none> and password file <none>
I20240503 08:00:22.199115 15330 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:22.199168 15330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:22.199339 15330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:22.200826 15330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8da39dc143464b32bc5b822a7abbafba"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.201282 15330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8da39dc143464b32bc5b822a7abbafba"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.204221 15330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 08:00:22.206166 15344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.207034 15330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:22.207140 15330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8da39dc143464b32bc5b822a7abbafba"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.207214 15330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:22.242028 15330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:22.242674 15330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:22.242825 15330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:22.248788 15330 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:34401
I20240503 08:00:22.248800 15396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:34401 every 8 connection(s)
I20240503 08:00:22.249660 15330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:00:22.251912 15397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:22.254262 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 15330
I20240503 08:00:22.254370 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:00:22.256582 15397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba: Bootstrap starting.
I20240503 08:00:22.258596 15397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:22.259212 15397 log.cc:824] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:22.260766 15397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba: No bootstrap required, opened a new log
I20240503 08:00:22.262996 15397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da39dc143464b32bc5b822a7abbafba" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34401 } }
I20240503 08:00:22.263159 15397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:22.263211 15397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da39dc143464b32bc5b822a7abbafba, State: Initialized, Role: FOLLOWER
I20240503 08:00:22.263582 15397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da39dc143464b32bc5b822a7abbafba" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34401 } }
I20240503 08:00:22.263695 15397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:22.263747 15397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:22.263840 15397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:22.264457 15397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da39dc143464b32bc5b822a7abbafba" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34401 } }
I20240503 08:00:22.264732 15397 leader_election.cc:302] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8da39dc143464b32bc5b822a7abbafba; no voters:
I20240503 08:00:22.264947 15397 leader_election.cc:288] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:22.265034 15402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:22.265250 15402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [term 1 LEADER]: Becoming Leader. State: Replica: 8da39dc143464b32bc5b822a7abbafba, State: Running, Role: LEADER
I20240503 08:00:22.265498 15402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da39dc143464b32bc5b822a7abbafba" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34401 } }
I20240503 08:00:22.265875 15397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:00:22.267177 15403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8da39dc143464b32bc5b822a7abbafba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da39dc143464b32bc5b822a7abbafba" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34401 } } }
I20240503 08:00:22.267200 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da39dc143464b32bc5b822a7abbafba. Latest consensus state: current_term: 1 leader_uuid: "8da39dc143464b32bc5b822a7abbafba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da39dc143464b32bc5b822a7abbafba" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34401 } } }
I20240503 08:00:22.267298 15403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:22.267298 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:22.267633 15416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:00:22.270797 15416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:00:22.275383 15416 catalog_manager.cc:1347] Generated new cluster ID: f063449e01c84a50b1916e611e0baac7
I20240503 08:00:22.275445 15416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:00:22.308974 15416 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:00:22.310134 15416 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:00:22.322386 15416 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8da39dc143464b32bc5b822a7abbafba: Generated new TSK 0
I20240503 08:00:22.322976 15416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:00:22.334750 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34401
--builtin_ntp_servers=127.0.99.148:41421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:00:22.453276 15421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:22.453452 15421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:22.453491 15421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:22.456595 15421 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:22.456650 15421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:22.456712 15421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 08:00:22.460160 15421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34401
--never_fsync=true
--heap_profile_path=/tmp/kudu.15421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:22.460870 15421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:22.461858 15421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:22.470124 15426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.470652 15427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.471635 15429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:22.472290 15421 server_base.cc:1034] running on GCE node
I20240503 08:00:22.472622 15421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:22.473170 15421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:22.474309 15421 hybrid_clock.cc:648] HybridClock initialized: now 1714723222474298 us; error 25 us; skew 500 ppm
I20240503 08:00:22.475885 15421 webserver.cc:460] Webserver started at http://127.0.99.129:34651/ using document root <none> and password file <none>
I20240503 08:00:22.476398 15421 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:22.476450 15421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:22.476620 15421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:22.478165 15421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.478579 15421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.481626 15421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:00:22.483587 15435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.484498 15421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:22.484598 15421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.484670 15421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:22.497623 15421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:22.498172 15421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:22.498312 15421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:22.498767 15421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:22.499581 15421 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:22.499625 15421 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.499675 15421 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:22.499698 15421 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.507411 15421 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:35409
I20240503 08:00:22.507445 15548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:35409 every 8 connection(s)
I20240503 08:00:22.508212 15421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:00:22.510308 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 15421
I20240503 08:00:22.510442 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:00:22.512348 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34401
--builtin_ntp_servers=127.0.99.148:41421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:22.516180 15549 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34401
I20240503 08:00:22.516395 15549 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:22.516983 15549 heartbeater.cc:511] Master 127.0.99.190:34401 requested a full tablet report, sending...
I20240503 08:00:22.554687 15361 ts_manager.cc:194] Registered new tserver with Master: fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:22.556473 15361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:53309
W20240503 08:00:22.618009 15552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:22.618201 15552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:22.618245 15552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:22.621150 15552 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:22.621205 15552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:22.621273 15552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 08:00:22.624655 15552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34401
--never_fsync=true
--heap_profile_path=/tmp/kudu.15552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:22.625430 15552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:22.626416 15552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:22.634440 15561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.635305 15559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.636049 15558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:22.636842 15552 server_base.cc:1034] running on GCE node
I20240503 08:00:22.637236 15552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:22.637764 15552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:22.638947 15552 hybrid_clock.cc:648] HybridClock initialized: now 1714723222638928 us; error 36 us; skew 500 ppm
I20240503 08:00:22.640763 15552 webserver.cc:460] Webserver started at http://127.0.99.130:38217/ using document root <none> and password file <none>
I20240503 08:00:22.641414 15552 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:22.641480 15552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:22.641662 15552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:22.643631 15552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0bfedb67806a457188db76fa09341dfd"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.644146 15552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0bfedb67806a457188db76fa09341dfd"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.647680 15552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20240503 08:00:22.649659 15567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.650624 15552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:22.650727 15552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0bfedb67806a457188db76fa09341dfd"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.650811 15552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:22.671841 15552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:22.672415 15552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:22.672551 15552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:22.673079 15552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:22.673915 15552 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:22.673960 15552 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.674010 15552 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:22.674037 15552 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.681851 15552 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:38965
I20240503 08:00:22.681890 15680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:38965 every 8 connection(s)
I20240503 08:00:22.682686 15552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:00:22.687835 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 15552
I20240503 08:00:22.687952 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:00:22.690229 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34401
--builtin_ntp_servers=127.0.99.148:41421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:22.690847 15681 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34401
I20240503 08:00:22.691040 15681 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:22.691561 15681 heartbeater.cc:511] Master 127.0.99.190:34401 requested a full tablet report, sending...
I20240503 08:00:22.725991 15361 ts_manager.cc:194] Registered new tserver with Master: 0bfedb67806a457188db76fa09341dfd (127.0.99.130:38965)
I20240503 08:00:22.726661 15361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:39323
W20240503 08:00:22.790410 15684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:22.790608 15684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:22.790665 15684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:22.793551 15684 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:22.793608 15684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:22.793687 15684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:00:22.797034 15684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34401
--never_fsync=true
--heap_profile_path=/tmp/kudu.15684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:22.797829 15684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:22.798813 15684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:22.806802 15690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.807657 15693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.808439 15691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:22.809233 15684 server_base.cc:1034] running on GCE node
I20240503 08:00:22.809568 15684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:22.810112 15684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:22.811256 15684 hybrid_clock.cc:648] HybridClock initialized: now 1714723222811248 us; error 29 us; skew 500 ppm
I20240503 08:00:22.812814 15684 webserver.cc:460] Webserver started at http://127.0.99.131:39763/ using document root <none> and password file <none>
I20240503 08:00:22.813334 15684 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:22.813385 15684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:22.813529 15684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:22.815033 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6be8cb3537d84305a11b3a8bf04a9026"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.815429 15684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6be8cb3537d84305a11b3a8bf04a9026"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.818359 15684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 08:00:22.820204 15699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.821125 15684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:22.821205 15684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6be8cb3537d84305a11b3a8bf04a9026"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.821270 15684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:22.831310 15684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:22.831749 15684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:22.831861 15684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:22.832257 15684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:22.832986 15684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:22.833027 15684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.833118 15684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:22.833144 15684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.840695 15684 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34211
I20240503 08:00:22.840741 15812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34211 every 8 connection(s)
I20240503 08:00:22.841522 15684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:00:22.845177 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 15684
I20240503 08:00:22.845268 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:00:22.847039 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34401
--builtin_ntp_servers=127.0.99.148:41421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:22.849637 15813 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34401
I20240503 08:00:22.849823 15813 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:22.850266 15813 heartbeater.cc:511] Master 127.0.99.190:34401 requested a full tablet report, sending...
I20240503 08:00:22.883423 15361 ts_manager.cc:194] Registered new tserver with Master: 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131:34211)
I20240503 08:00:22.884428 15361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:59141
W20240503 08:00:22.951212 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:22.951470 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:22.951515 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:22.954488 15816 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:22.954555 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:22.954623 15816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:00:22.958140 15816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34401
--never_fsync=true
--heap_profile_path=/tmp/kudu.15816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:22.958935 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:22.959997 15816 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:22.968780 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.969300 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:22.970229 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:22.970861 15816 server_base.cc:1034] running on GCE node
I20240503 08:00:22.971211 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:22.971839 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:22.972987 15816 hybrid_clock.cc:648] HybridClock initialized: now 1714723222972975 us; error 28 us; skew 500 ppm
I20240503 08:00:22.974665 15816 webserver.cc:460] Webserver started at http://127.0.99.132:37593/ using document root <none> and password file <none>
I20240503 08:00:22.975170 15816 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:22.975225 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:22.975384 15816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:22.976933 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.977413 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.980486 15816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:00:22.982529 15831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:22.983568 15816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:22.983669 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:22.983749 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:23.010071 15816 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:23.010674 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:23.010828 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:23.011338 15816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:23.012219 15816 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:23.012264 15816 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:23.012302 15816 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:23.012326 15816 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:23.020153 15816 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37461
I20240503 08:00:23.020176 15944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37461 every 8 connection(s)
I20240503 08:00:23.021162 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:00:23.022370 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 15816
I20240503 08:00:23.022481 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:00:23.029625 15945 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34401
I20240503 08:00:23.029824 15945 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:23.030320 15945 heartbeater.cc:511] Master 127.0.99.190:34401 requested a full tablet report, sending...
I20240503 08:00:23.062896 15361 ts_manager.cc:194] Registered new tserver with Master: 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:23.063572 15361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:56153
I20240503 08:00:23.073192 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:00:23.082204 398 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 08:00:23.082284 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 15552
I20240503 08:00:23.087512 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 15684
I20240503 08:00:23.558893 15549 heartbeater.cc:503] Master 127.0.99.190:34401 was elected leader, sending a full tablet report...
I20240503 08:00:24.065887 15945 heartbeater.cc:503] Master 127.0.99.190:34401 was elected leader, sending a full tablet report...
I20240503 08:00:26.843431 15361 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48042:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:26.846411 15361 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 08:00:26.861035 15482 tablet_service.cc:1467] Processing CreateTablet for tablet 966180cb6070430bafa6f04e5753d518 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a86ef2e7fcf8450fa5f40ab84054bf6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:26.861037 15878 tablet_service.cc:1467] Processing CreateTablet for tablet 966180cb6070430bafa6f04e5753d518 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a86ef2e7fcf8450fa5f40ab84054bf6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:26.861037 15483 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a86ef2e7fcf8450fa5f40ab84054bf6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:26.861188 15481 tablet_service.cc:1467] Processing CreateTablet for tablet b83628c5f5f9459d88defa892f7a7fab (DEFAULT_TABLE table=rebalance_test_table_0 [id=a86ef2e7fcf8450fa5f40ab84054bf6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:26.861974 15483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.861974 15878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966180cb6070430bafa6f04e5753d518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.862170 15877 tablet_service.cc:1467] Processing CreateTablet for tablet b83628c5f5f9459d88defa892f7a7fab (DEFAULT_TABLE table=rebalance_test_table_0 [id=a86ef2e7fcf8450fa5f40ab84054bf6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:26.862293 15482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966180cb6070430bafa6f04e5753d518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.862390 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83628c5f5f9459d88defa892f7a7fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.861035 15879 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a86ef2e7fcf8450fa5f40ab84054bf6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:26.863039 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.865147 15481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83628c5f5f9459d88defa892f7a7fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:26.869043 15962 tablet_bootstrap.cc:492] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487: Bootstrap starting.
I20240503 08:00:26.869146 15963 tablet_bootstrap.cc:492] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Bootstrap starting.
I20240503 08:00:26.871009 15963 tablet_bootstrap.cc:654] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:26.871009 15962 tablet_bootstrap.cc:654] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:26.871850 15962 log.cc:824] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:26.871801 15963 log.cc:824] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:26.873224 15962 tablet_bootstrap.cc:492] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487: No bootstrap required, opened a new log
I20240503 08:00:26.873251 15963 tablet_bootstrap.cc:492] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: No bootstrap required, opened a new log
I20240503 08:00:26.873371 15962 ts_tablet_manager.cc:1367] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 08:00:26.873374 15963 ts_tablet_manager.cc:1367] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:00:26.875882 15962 raft_consensus.cc:348] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.876036 15962 raft_consensus.cc:374] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:26.876086 15962 raft_consensus.cc:729] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.876044 15963 raft_consensus.cc:348] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.876214 15963 raft_consensus.cc:374] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:26.876307 15963 raft_consensus.cc:729] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.876473 15962 consensus_queue.cc:260] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.876647 15963 consensus_queue.cc:260] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.877238 15962 ts_tablet_manager.cc:1398] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240503 08:00:26.877377 15963 ts_tablet_manager.cc:1398] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:00:26.877462 15962 tablet_bootstrap.cc:492] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487: Bootstrap starting.
I20240503 08:00:26.877995 15963 tablet_bootstrap.cc:492] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Bootstrap starting.
I20240503 08:00:26.878276 15962 tablet_bootstrap.cc:654] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:26.879429 15963 tablet_bootstrap.cc:654] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:26.879500 15962 tablet_bootstrap.cc:492] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487: No bootstrap required, opened a new log
I20240503 08:00:26.879614 15962 ts_tablet_manager.cc:1367] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:26.880116 15962 raft_consensus.cc:348] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.880209 15962 raft_consensus.cc:374] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:26.880240 15962 raft_consensus.cc:729] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.880406 15962 consensus_queue.cc:260] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.880589 15963 tablet_bootstrap.cc:492] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: No bootstrap required, opened a new log
I20240503 08:00:26.880687 15963 ts_tablet_manager.cc:1367] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:26.880697 15962 ts_tablet_manager.cc:1398] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:26.880887 15962 tablet_bootstrap.cc:492] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487: Bootstrap starting.
I20240503 08:00:26.881361 15963 raft_consensus.cc:348] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.881436 15963 raft_consensus.cc:374] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:26.881464 15963 raft_consensus.cc:729] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.881620 15963 consensus_queue.cc:260] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.881705 15962 tablet_bootstrap.cc:654] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:26.881896 15963 ts_tablet_manager.cc:1398] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:26.882047 15963 tablet_bootstrap.cc:492] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Bootstrap starting.
I20240503 08:00:26.882730 15962 tablet_bootstrap.cc:492] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487: No bootstrap required, opened a new log
I20240503 08:00:26.882809 15963 tablet_bootstrap.cc:654] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:26.882850 15962 ts_tablet_manager.cc:1367] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:26.883262 15962 raft_consensus.cc:348] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.883353 15962 raft_consensus.cc:374] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:26.883411 15962 raft_consensus.cc:729] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.883530 15962 consensus_queue.cc:260] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.883831 15963 tablet_bootstrap.cc:492] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5: No bootstrap required, opened a new log
I20240503 08:00:26.883828 15962 ts_tablet_manager.cc:1398] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:26.883942 15963 ts_tablet_manager.cc:1367] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:26.884311 15963 raft_consensus.cc:348] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.884384 15963 raft_consensus.cc:374] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:26.884413 15963 raft_consensus.cc:729] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Initialized, Role: FOLLOWER
I20240503 08:00:26.884516 15963 consensus_queue.cc:260] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.884755 15963 ts_tablet_manager.cc:1398] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:26.936362 15966 raft_consensus.cc:482] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:26.936540 15966 raft_consensus.cc:504] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.937522 15966 leader_election.cc:288] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:26.940925 15899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83628c5f5f9459d88defa892f7a7fab" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" is_pre_election: true
I20240503 08:00:26.941181 15899 raft_consensus.cc:2454] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 0.
I20240503 08:00:26.941565 15438 leader_election.cc:302] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:26.941781 15966 raft_consensus.cc:2781] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:26.941851 15966 raft_consensus.cc:482] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:26.941897 15966 raft_consensus.cc:3037] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:26.942703 15966 raft_consensus.cc:504] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.942937 15966 leader_election.cc:288] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Requested vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:26.943272 15899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b83628c5f5f9459d88defa892f7a7fab" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
I20240503 08:00:26.943364 15899 raft_consensus.cc:3037] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:26.944372 15899 raft_consensus.cc:2454] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 1.
I20240503 08:00:26.944629 15438 leader_election.cc:302] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:26.944787 15966 raft_consensus.cc:2781] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:26.944967 15966 raft_consensus.cc:686] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Becoming Leader. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Running, Role: LEADER
I20240503 08:00:26.945261 15966 consensus_queue.cc:237] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:26.947873 15361 catalog_manager.cc:5484] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:27.011932 15550 tablet.cc:2369] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:27.024441 15946 tablet.cc:2369] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:00:27.155505 15967 raft_consensus.cc:482] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:27.155640 15967 raft_consensus.cc:504] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.156531 15967 leader_election.cc:288] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:27.159719 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966180cb6070430bafa6f04e5753d518" candidate_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" is_pre_election: true
I20240503 08:00:27.159929 15503 raft_consensus.cc:2454] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2703c9a1353f4e3c9f8bbb7b5e52eab5 in term 0.
I20240503 08:00:27.160297 15833 leader_election.cc:302] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.160524 15967 raft_consensus.cc:2781] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:27.160593 15967 raft_consensus.cc:482] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:27.160637 15967 raft_consensus.cc:3037] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.161437 15967 raft_consensus.cc:504] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.161684 15967 leader_election.cc:288] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 election: Requested vote from peers fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:27.162027 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966180cb6070430bafa6f04e5753d518" candidate_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
I20240503 08:00:27.162138 15503 raft_consensus.cc:3037] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.162911 15503 raft_consensus.cc:2454] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703c9a1353f4e3c9f8bbb7b5e52eab5 in term 1.
I20240503 08:00:27.163158 15833 leader_election.cc:302] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.163321 15967 raft_consensus.cc:2781] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:27.163502 15967 raft_consensus.cc:686] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 LEADER]: Becoming Leader. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Running, Role: LEADER
I20240503 08:00:27.163786 15967 consensus_queue.cc:237] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.166406 15361 catalog_manager.cc:5484] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:27.178633 15967 raft_consensus.cc:482] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:27.178705 15967 raft_consensus.cc:504] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.178933 15967 leader_election.cc:288] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:27.179240 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6c3ce2ccbc42fbbd66c578aa40e1f4" candidate_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" is_pre_election: true
I20240503 08:00:27.179345 15503 raft_consensus.cc:2454] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2703c9a1353f4e3c9f8bbb7b5e52eab5 in term 0.
I20240503 08:00:27.179577 15833 leader_election.cc:302] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.179716 15967 raft_consensus.cc:2781] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:27.179764 15967 raft_consensus.cc:482] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:27.179805 15967 raft_consensus.cc:3037] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.180416 15967 raft_consensus.cc:504] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.180634 15967 leader_election.cc:288] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 election: Requested vote from peers fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:27.180887 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6c3ce2ccbc42fbbd66c578aa40e1f4" candidate_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
I20240503 08:00:27.180974 15503 raft_consensus.cc:3037] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.181614 15503 raft_consensus.cc:2454] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703c9a1353f4e3c9f8bbb7b5e52eab5 in term 1.
I20240503 08:00:27.181859 15833 leader_election.cc:302] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.182008 15967 raft_consensus.cc:2781] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:27.182103 15967 raft_consensus.cc:686] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 LEADER]: Becoming Leader. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Running, Role: LEADER
I20240503 08:00:27.182211 15967 consensus_queue.cc:237] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.183437 15361 catalog_manager.cc:5484] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } }
W20240503 08:00:27.305850 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:27.306053 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:27.308909 15979 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:00:27.402602 15503 raft_consensus.cc:1261] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:27.403450 15967 consensus_queue.cc:1035] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:27.405622 15503 raft_consensus.cc:1261] T 966180cb6070430bafa6f04e5753d518 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:27.406069 15967 consensus_queue.cc:1035] T 966180cb6070430bafa6f04e5753d518 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:27.407693 15899 raft_consensus.cc:1261] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:27.409253 15966 consensus_queue.cc:1035] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:27.411408 16006 mvcc.cc:205] Tried to move back new op lower bound from 7023506339434860544 to 7023506338539040768. Current Snapshot: MvccSnapshot[applied={T|T < 7023506339434860544}]
I20240503 08:00:27.415831 16011 mvcc.cc:205] Tried to move back new op lower bound from 7023506339452284928 to 7023506337569406976. Current Snapshot: MvccSnapshot[applied={T|T < 7023506339452284928}]
INSERT report
rows total: 2000
time total: 95.4588 ms
time per row: 0.0477294 ms
I20240503 08:00:27.443778 15358 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48042:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:27.444231 15358 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 08:00:27.448489 15879 tablet_service.cc:1467] Processing CreateTablet for tablet 40cc8a45aece478bb35df85206a99b0f (DEFAULT_TABLE table=rebalance_test_table_1 [id=75b65d480fb4494faa4a430e500f8f2e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:27.448485 15482 tablet_service.cc:1467] Processing CreateTablet for tablet 40cc8a45aece478bb35df85206a99b0f (DEFAULT_TABLE table=rebalance_test_table_1 [id=75b65d480fb4494faa4a430e500f8f2e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:27.448740 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40cc8a45aece478bb35df85206a99b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:27.448740 15482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40cc8a45aece478bb35df85206a99b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:27.450414 16027 tablet_bootstrap.cc:492] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487: Bootstrap starting.
I20240503 08:00:27.450493 16026 tablet_bootstrap.cc:492] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Bootstrap starting.
I20240503 08:00:27.450764 15877 tablet_service.cc:1467] Processing CreateTablet for tablet bc0913594afc4641a8f233e28a5f3c50 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75b65d480fb4494faa4a430e500f8f2e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:27.450974 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0913594afc4641a8f233e28a5f3c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:27.451226 16027 tablet_bootstrap.cc:654] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:27.451207 15483 tablet_service.cc:1467] Processing CreateTablet for tablet bc0913594afc4641a8f233e28a5f3c50 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75b65d480fb4494faa4a430e500f8f2e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:27.451406 15483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0913594afc4641a8f233e28a5f3c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:27.452162 16027 tablet_bootstrap.cc:492] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487: No bootstrap required, opened a new log
I20240503 08:00:27.452229 16027 ts_tablet_manager.cc:1367] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:27.452343 16026 tablet_bootstrap.cc:654] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:27.452680 16027 raft_consensus.cc:348] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.452797 16027 raft_consensus.cc:374] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:27.452847 16027 raft_consensus.cc:729] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Initialized, Role: FOLLOWER
I20240503 08:00:27.452987 16027 consensus_queue.cc:260] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.453002 15482 tablet_service.cc:1467] Processing CreateTablet for tablet f1f7a626b7a044d9918ff22e2f7bf583 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75b65d480fb4494faa4a430e500f8f2e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:27.453207 15482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f7a626b7a044d9918ff22e2f7bf583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:27.453265 16027 ts_tablet_manager.cc:1398] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:27.453400 16027 tablet_bootstrap.cc:492] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487: Bootstrap starting.
I20240503 08:00:27.454331 16027 tablet_bootstrap.cc:654] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:27.454650 15879 tablet_service.cc:1467] Processing CreateTablet for tablet f1f7a626b7a044d9918ff22e2f7bf583 (DEFAULT_TABLE table=rebalance_test_table_1 [id=75b65d480fb4494faa4a430e500f8f2e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:27.454835 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f7a626b7a044d9918ff22e2f7bf583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:27.455294 16027 tablet_bootstrap.cc:492] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487: No bootstrap required, opened a new log
I20240503 08:00:27.455411 16027 ts_tablet_manager.cc:1367] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:27.455602 16026 tablet_bootstrap.cc:492] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5: No bootstrap required, opened a new log
I20240503 08:00:27.455682 16026 ts_tablet_manager.cc:1367] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:00:27.455811 16027 raft_consensus.cc:348] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:27.455897 16027 raft_consensus.cc:374] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:27.455955 16027 raft_consensus.cc:729] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Initialized, Role: FOLLOWER
I20240503 08:00:27.456070 16027 consensus_queue.cc:260] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:27.456082 16026 raft_consensus.cc:348] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.456157 16026 raft_consensus.cc:374] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:27.456192 16026 raft_consensus.cc:729] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Initialized, Role: FOLLOWER
I20240503 08:00:27.456300 16026 consensus_queue.cc:260] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.456374 16027 ts_tablet_manager.cc:1398] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:27.456506 16027 tablet_bootstrap.cc:492] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487: Bootstrap starting.
I20240503 08:00:27.456527 16026 ts_tablet_manager.cc:1398] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:27.456656 16026 tablet_bootstrap.cc:492] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Bootstrap starting.
I20240503 08:00:27.457310 16027 tablet_bootstrap.cc:654] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:27.457649 16026 tablet_bootstrap.cc:654] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:27.458405 16027 tablet_bootstrap.cc:492] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487: No bootstrap required, opened a new log
I20240503 08:00:27.458520 16027 ts_tablet_manager.cc:1367] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:27.458921 16027 raft_consensus.cc:348] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.458997 16027 raft_consensus.cc:374] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:27.459015 16027 raft_consensus.cc:729] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Initialized, Role: FOLLOWER
I20240503 08:00:27.459070 16026 tablet_bootstrap.cc:492] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: No bootstrap required, opened a new log
I20240503 08:00:27.459100 16027 consensus_queue.cc:260] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.459144 16026 ts_tablet_manager.cc:1367] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:27.459335 16027 ts_tablet_manager.cc:1398] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:27.459486 16026 raft_consensus.cc:348] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:27.459575 16026 raft_consensus.cc:374] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:27.459626 16026 raft_consensus.cc:729] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Initialized, Role: FOLLOWER
I20240503 08:00:27.459753 16026 consensus_queue.cc:260] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:27.460008 16026 ts_tablet_manager.cc:1398] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:27.460160 16026 tablet_bootstrap.cc:492] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Bootstrap starting.
I20240503 08:00:27.460903 16026 tablet_bootstrap.cc:654] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:27.461855 16026 tablet_bootstrap.cc:492] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: No bootstrap required, opened a new log
I20240503 08:00:27.461930 16026 ts_tablet_manager.cc:1367] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:27.462297 16026 raft_consensus.cc:348] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.462389 16026 raft_consensus.cc:374] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:27.462432 16026 raft_consensus.cc:729] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Initialized, Role: FOLLOWER
I20240503 08:00:27.462536 16026 consensus_queue.cc:260] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.462749 16026 ts_tablet_manager.cc:1398] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:27.522354 16022 raft_consensus.cc:482] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:27.522472 16022 raft_consensus.cc:504] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.522801 16022 leader_election.cc:288] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:27.523182 15899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40cc8a45aece478bb35df85206a99b0f" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" is_pre_election: true
I20240503 08:00:27.523312 15899 raft_consensus.cc:2454] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 0.
I20240503 08:00:27.523579 15438 leader_election.cc:302] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.523730 16022 raft_consensus.cc:2781] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:27.523797 16022 raft_consensus.cc:482] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:27.523831 16022 raft_consensus.cc:3037] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.524798 16022 raft_consensus.cc:504] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.525089 16022 leader_election.cc:288] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Requested vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:27.525403 15899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40cc8a45aece478bb35df85206a99b0f" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
I20240503 08:00:27.525532 15899 raft_consensus.cc:3037] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.526268 15899 raft_consensus.cc:2454] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 1.
I20240503 08:00:27.526491 15438 leader_election.cc:302] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.526664 16022 raft_consensus.cc:2781] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:27.526741 16022 raft_consensus.cc:686] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Becoming Leader. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Running, Role: LEADER
I20240503 08:00:27.526842 16022 consensus_queue.cc:237] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.528326 15357 catalog_manager.cc:5484] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:27.606485 16022 raft_consensus.cc:482] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:27.606599 16022 raft_consensus.cc:504] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:27.606912 16022 leader_election.cc:288] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:27.607278 15899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0913594afc4641a8f233e28a5f3c50" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" is_pre_election: true
I20240503 08:00:27.607405 15899 raft_consensus.cc:2454] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 0.
I20240503 08:00:27.607648 15438 leader_election.cc:302] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.607815 16022 raft_consensus.cc:2781] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:27.607868 16022 raft_consensus.cc:482] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:27.607903 16022 raft_consensus.cc:3037] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.608659 16022 raft_consensus.cc:504] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:27.608891 16022 leader_election.cc:288] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Requested vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:27.609293 15899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0913594afc4641a8f233e28a5f3c50" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
I20240503 08:00:27.609406 15899 raft_consensus.cc:3037] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.610162 15899 raft_consensus.cc:2454] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 1.
I20240503 08:00:27.610441 15438 leader_election.cc:302] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.610597 16022 raft_consensus.cc:2781] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:27.610688 16022 raft_consensus.cc:686] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Becoming Leader. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Running, Role: LEADER
I20240503 08:00:27.610817 16022 consensus_queue.cc:237] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:27.612165 15357 catalog_manager.cc:5484] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:27.786381 15967 raft_consensus.cc:482] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:27.786495 15967 raft_consensus.cc:504] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.786808 15967 leader_election.cc:288] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:27.787221 15502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f7a626b7a044d9918ff22e2f7bf583" candidate_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" is_pre_election: true
I20240503 08:00:27.787354 15502 raft_consensus.cc:2454] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2703c9a1353f4e3c9f8bbb7b5e52eab5 in term 0.
I20240503 08:00:27.787648 15833 leader_election.cc:302] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.787825 15967 raft_consensus.cc:2781] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:27.787881 15967 raft_consensus.cc:482] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:27.787909 15967 raft_consensus.cc:3037] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.788693 15967 raft_consensus.cc:504] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.788923 15967 leader_election.cc:288] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 election: Requested vote from peers fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:27.789355 15502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f7a626b7a044d9918ff22e2f7bf583" candidate_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
I20240503 08:00:27.789451 15502 raft_consensus.cc:3037] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:27.790143 15502 raft_consensus.cc:2454] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703c9a1353f4e3c9f8bbb7b5e52eab5 in term 1.
I20240503 08:00:27.790405 15833 leader_election.cc:302] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:27.790575 15967 raft_consensus.cc:2781] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:27.790656 15967 raft_consensus.cc:686] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 LEADER]: Becoming Leader. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Running, Role: LEADER
I20240503 08:00:27.790782 15967 consensus_queue.cc:237] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:27.792095 15357 catalog_manager.cc:5484] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } }
W20240503 08:00:27.903532 16037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:27.903740 16037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:27.906630 16037 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:00:27.977643 15899 raft_consensus.cc:1261] T bc0913594afc4641a8f233e28a5f3c50 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:27.978008 16022 consensus_queue.cc:1035] T bc0913594afc4641a8f233e28a5f3c50 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:27.980031 15898 raft_consensus.cc:1261] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:27.980518 16022 consensus_queue.cc:1035] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:27.980572 15502 raft_consensus.cc:1261] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:27.981108 15967 consensus_queue.cc:1035] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 78.9271 ms
time per row: 0.0394636 ms
I20240503 08:00:28.021731 15360 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48042:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:28.022198 15360 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240503 08:00:28.027128 15482 tablet_service.cc:1467] Processing CreateTablet for tablet b4b2d66e4b4147888ee7a3a953d24bee (DEFAULT_TABLE table=rebalance_test_table_2 [id=050c272c47d64b5498447aaf07125871]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:28.027174 15483 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4273eac3c484fa7e7af6bf0392702 (DEFAULT_TABLE table=rebalance_test_table_2 [id=050c272c47d64b5498447aaf07125871]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:28.027432 15482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b2d66e4b4147888ee7a3a953d24bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:28.027701 15877 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4273eac3c484fa7e7af6bf0392702 (DEFAULT_TABLE table=rebalance_test_table_2 [id=050c272c47d64b5498447aaf07125871]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:28.027906 15878 tablet_service.cc:1467] Processing CreateTablet for tablet 5a823284845c4f70b643ef1e8e187282 (DEFAULT_TABLE table=rebalance_test_table_2 [id=050c272c47d64b5498447aaf07125871]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:28.027976 15483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4273eac3c484fa7e7af6bf0392702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:28.027716 15879 tablet_service.cc:1467] Processing CreateTablet for tablet b4b2d66e4b4147888ee7a3a953d24bee (DEFAULT_TABLE table=rebalance_test_table_2 [id=050c272c47d64b5498447aaf07125871]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:28.027921 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4273eac3c484fa7e7af6bf0392702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:28.029147 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b2d66e4b4147888ee7a3a953d24bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:28.029680 15481 tablet_service.cc:1467] Processing CreateTablet for tablet 5a823284845c4f70b643ef1e8e187282 (DEFAULT_TABLE table=rebalance_test_table_2 [id=050c272c47d64b5498447aaf07125871]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:28.029870 15481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a823284845c4f70b643ef1e8e187282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:28.030988 16086 tablet_bootstrap.cc:492] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Bootstrap starting.
I20240503 08:00:28.031075 15878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a823284845c4f70b643ef1e8e187282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:28.031988 16086 tablet_bootstrap.cc:654] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:28.033409 16086 tablet_bootstrap.cc:492] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5: No bootstrap required, opened a new log
I20240503 08:00:28.033515 16086 ts_tablet_manager.cc:1367] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:28.033903 16085 tablet_bootstrap.cc:492] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487: Bootstrap starting.
I20240503 08:00:28.033912 16086 raft_consensus.cc:348] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:28.034035 16086 raft_consensus.cc:374] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:28.034085 16086 raft_consensus.cc:729] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Initialized, Role: FOLLOWER
I20240503 08:00:28.034209 16086 consensus_queue.cc:260] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:28.034533 16086 ts_tablet_manager.cc:1398] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:28.034684 16086 tablet_bootstrap.cc:492] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Bootstrap starting.
I20240503 08:00:28.034693 16085 tablet_bootstrap.cc:654] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:28.035432 16086 tablet_bootstrap.cc:654] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:28.035984 16085 tablet_bootstrap.cc:492] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487: No bootstrap required, opened a new log
I20240503 08:00:28.036064 16085 ts_tablet_manager.cc:1367] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:28.036478 16085 raft_consensus.cc:348] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:28.036568 16085 raft_consensus.cc:374] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:28.036599 16085 raft_consensus.cc:729] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Initialized, Role: FOLLOWER
I20240503 08:00:28.036710 16085 consensus_queue.cc:260] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:28.036964 16085 ts_tablet_manager.cc:1398] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:28.037457 16085 tablet_bootstrap.cc:492] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: Bootstrap starting.
I20240503 08:00:28.038117 16085 tablet_bootstrap.cc:654] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:28.038966 16085 tablet_bootstrap.cc:492] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: No bootstrap required, opened a new log
I20240503 08:00:28.039036 16085 ts_tablet_manager.cc:1367] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:28.039382 16085 raft_consensus.cc:348] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.039465 16085 raft_consensus.cc:374] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:28.039533 16085 raft_consensus.cc:729] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Initialized, Role: FOLLOWER
I20240503 08:00:28.039654 16085 consensus_queue.cc:260] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.040175 16085 ts_tablet_manager.cc:1398] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:28.040326 16085 tablet_bootstrap.cc:492] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487: Bootstrap starting.
I20240503 08:00:28.041116 16085 tablet_bootstrap.cc:654] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:28.041779 16086 tablet_bootstrap.cc:492] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: No bootstrap required, opened a new log
I20240503 08:00:28.041893 16086 ts_tablet_manager.cc:1367] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:00:28.042064 16085 tablet_bootstrap.cc:492] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487: No bootstrap required, opened a new log
I20240503 08:00:28.042137 16085 ts_tablet_manager.cc:1367] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:28.042307 16086 raft_consensus.cc:348] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.042382 16086 raft_consensus.cc:374] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:28.042404 16086 raft_consensus.cc:729] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Initialized, Role: FOLLOWER
I20240503 08:00:28.042490 16086 consensus_queue.cc:260] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.042479 16085 raft_consensus.cc:348] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.042570 16085 raft_consensus.cc:374] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:28.042603 16085 raft_consensus.cc:729] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Initialized, Role: FOLLOWER
I20240503 08:00:28.042698 16085 consensus_queue.cc:260] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.042937 16085 ts_tablet_manager.cc:1398] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:28.043197 16086 ts_tablet_manager.cc:1398] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:28.043382 16086 tablet_bootstrap.cc:492] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Bootstrap starting.
I20240503 08:00:28.045257 16086 tablet_bootstrap.cc:654] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:28.046157 16086 tablet_bootstrap.cc:492] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: No bootstrap required, opened a new log
I20240503 08:00:28.046227 16086 ts_tablet_manager.cc:1367] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:28.046559 16086 raft_consensus.cc:348] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.046650 16086 raft_consensus.cc:374] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:28.046684 16086 raft_consensus.cc:729] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703c9a1353f4e3c9f8bbb7b5e52eab5, State: Initialized, Role: FOLLOWER
I20240503 08:00:28.046806 16086 consensus_queue.cc:260] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.047082 16086 ts_tablet_manager.cc:1398] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:28.235611 16022 raft_consensus.cc:482] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:28.235744 16022 raft_consensus.cc:504] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.236088 16022 leader_election.cc:288] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:28.236507 15898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a823284845c4f70b643ef1e8e187282" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" is_pre_election: true
I20240503 08:00:28.236641 15898 raft_consensus.cc:2454] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 0.
I20240503 08:00:28.236927 15438 leader_election.cc:302] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:28.237149 16022 raft_consensus.cc:2781] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:28.237208 16022 raft_consensus.cc:482] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:28.237234 16022 raft_consensus.cc:3037] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:28.238003 16022 raft_consensus.cc:504] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.238246 16022 leader_election.cc:288] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Requested vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:28.238566 15898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a823284845c4f70b643ef1e8e187282" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
I20240503 08:00:28.238664 15898 raft_consensus.cc:3037] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:28.239668 15898 raft_consensus.cc:2454] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 1.
I20240503 08:00:28.239981 15438 leader_election.cc:302] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:28.240154 16022 raft_consensus.cc:2781] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:28.240240 16022 raft_consensus.cc:686] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Becoming Leader. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Running, Role: LEADER
I20240503 08:00:28.240361 16022 consensus_queue.cc:237] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.241804 15359 catalog_manager.cc:5484] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:28.274344 16022 raft_consensus.cc:482] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:28.274451 16022 raft_consensus.cc:504] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:28.274701 16022 leader_election.cc:288] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:28.275029 15898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b2d66e4b4147888ee7a3a953d24bee" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" is_pre_election: true
I20240503 08:00:28.275137 15898 raft_consensus.cc:2454] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 0.
I20240503 08:00:28.275359 15438 leader_election.cc:302] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:28.275560 16022 raft_consensus.cc:2781] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:28.275614 16022 raft_consensus.cc:482] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:28.275638 16022 raft_consensus.cc:3037] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:28.276465 16022 raft_consensus.cc:504] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:28.276734 16022 leader_election.cc:288] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Requested vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:28.277045 15898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b2d66e4b4147888ee7a3a953d24bee" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
I20240503 08:00:28.277179 15898 raft_consensus.cc:3037] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:28.277830 15898 raft_consensus.cc:2454] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 1.
I20240503 08:00:28.278067 15438 leader_election.cc:302] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:28.278203 16022 raft_consensus.cc:2781] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:28.278282 16022 raft_consensus.cc:686] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Becoming Leader. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Running, Role: LEADER
I20240503 08:00:28.278398 16022 consensus_queue.cc:237] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } }
I20240503 08:00:28.279744 15359 catalog_manager.cc:5484] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:28.323649 16022 raft_consensus.cc:482] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:28.323783 16022 raft_consensus.cc:504] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.324098 16022 leader_election.cc:288] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:28.324505 15898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4273eac3c484fa7e7af6bf0392702" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" is_pre_election: true
I20240503 08:00:28.324642 15898 raft_consensus.cc:2454] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 0.
I20240503 08:00:28.324918 15438 leader_election.cc:302] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:28.325101 16022 raft_consensus.cc:2781] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:28.325170 16022 raft_consensus.cc:482] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:28.325218 16022 raft_consensus.cc:3037] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:28.326040 16022 raft_consensus.cc:504] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.326310 16022 leader_election.cc:288] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Requested vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:28.326828 15898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4273eac3c484fa7e7af6bf0392702" candidate_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
I20240503 08:00:28.326941 15898 raft_consensus.cc:3037] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:28.327592 15898 raft_consensus.cc:2454] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec9c86608704e8e8fb88d3f6b6bf487 in term 1.
I20240503 08:00:28.327822 15438 leader_election.cc:302] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, fec9c86608704e8e8fb88d3f6b6bf487; no voters:
I20240503 08:00:28.327965 16022 raft_consensus.cc:2781] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:28.328049 16022 raft_consensus.cc:686] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Becoming Leader. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Running, Role: LEADER
I20240503 08:00:28.328171 16022 consensus_queue.cc:237] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } }
I20240503 08:00:28.329555 15359 catalog_manager.cc:5484] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } }
W20240503 08:00:28.488015 16096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:28.488234 16096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:28.491269 16096 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:00:28.560827 15898 raft_consensus.cc:1261] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:28.561280 16022 consensus_queue.cc:1035] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:28.561748 15898 raft_consensus.cc:1261] T 5a823284845c4f70b643ef1e8e187282 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:28.562152 16022 consensus_queue.cc:1035] T 5a823284845c4f70b643ef1e8e187282 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:28.563882 15897 raft_consensus.cc:1261] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:28.564335 16022 consensus_queue.cc:1035] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 73.7919 ms
time per row: 0.036896 ms
I20240503 08:00:28.603511 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:38965
--local_ip_for_outbound_sockets=127.0.99.130
--tserver_master_addrs=127.0.99.190:34401
--webserver_port=38217
--webserver_interface=127.0.99.130
--builtin_ntp_servers=127.0.99.148:41421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:00:28.705667 16140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:28.705863 16140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:28.705909 16140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:28.708798 16140 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:28.708853 16140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:28.708942 16140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 08:00:28.712610 16140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:38965
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=38217
--tserver_master_addrs=127.0.99.190:34401
--never_fsync=true
--heap_profile_path=/tmp/kudu.16140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:28.713441 16140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:28.714453 16140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:28.722580 16148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:28.723156 16146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:28.724115 16145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:28.724776 16140 server_base.cc:1034] running on GCE node
I20240503 08:00:28.725104 16140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:28.725589 16140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:28.726719 16140 hybrid_clock.cc:648] HybridClock initialized: now 1714723228726708 us; error 24 us; skew 500 ppm
I20240503 08:00:28.728224 16140 webserver.cc:460] Webserver started at http://127.0.99.130:38217/ using document root <none> and password file <none>
I20240503 08:00:28.728693 16140 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:28.728761 16140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:28.731695 16140 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:28.733345 16154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:28.734225 16140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:28.734316 16140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0bfedb67806a457188db76fa09341dfd"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:28.734671 16140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:28.746567 16140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:28.747040 16140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:28.747171 16140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:28.747599 16140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:28.748359 16140 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:28.748404 16140 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:28.748450 16140 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:28.748477 16140 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:28.756259 16140 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:38965
I20240503 08:00:28.756301 16267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:38965 every 8 connection(s)
I20240503 08:00:28.757169 16140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:00:28.758105 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 16140
I20240503 08:00:28.758447 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:34211
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:34401
--webserver_port=39763
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:41421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:28.765875 16268 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34401
I20240503 08:00:28.766091 16268 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:28.766638 16268 heartbeater.cc:511] Master 127.0.99.190:34401 requested a full tablet report, sending...
I20240503 08:00:28.767740 15359 ts_manager.cc:194] Re-registered known tserver with Master: 0bfedb67806a457188db76fa09341dfd (127.0.99.130:38965)
I20240503 08:00:28.768272 15359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:48859
W20240503 08:00:28.858592 16271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:28.858776 16271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:28.858831 16271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:28.861722 16271 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:28.861778 16271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:28.861852 16271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:00:28.865252 16271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:34211
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=39763
--tserver_master_addrs=127.0.99.190:34401
--never_fsync=true
--heap_profile_path=/tmp/kudu.16271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:28.865991 16271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:28.866999 16271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:28.875020 16279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:28.875501 16277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:28.876695 16276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:28.877357 16271 server_base.cc:1034] running on GCE node
I20240503 08:00:28.877722 16271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:28.878257 16271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:28.879418 16271 hybrid_clock.cc:648] HybridClock initialized: now 1714723228879405 us; error 25 us; skew 500 ppm
I20240503 08:00:28.881023 16271 webserver.cc:460] Webserver started at http://127.0.99.131:39763/ using document root <none> and password file <none>
I20240503 08:00:28.881525 16271 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:28.881592 16271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:28.884547 16271 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 08:00:28.886242 16285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:28.887151 16271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:28.887240 16271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6be8cb3537d84305a11b3a8bf04a9026"
format_stamp: "Formatted at 2024-05-03 08:00:22 on dist-test-slave-7b05"
I20240503 08:00:28.887629 16271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:28.900310 16271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:28.900802 16271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:28.900938 16271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:28.901412 16271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:28.902181 16271 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:28.902225 16271 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:28.902271 16271 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:28.902299 16271 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:28.909677 16271 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34211
I20240503 08:00:28.909715 16398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34211 every 8 connection(s)
I20240503 08:00:28.910600 16271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:00:28.913836 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 16271
I20240503 08:00:28.918856 16399 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34401
I20240503 08:00:28.919063 16399 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:28.919579 16399 heartbeater.cc:511] Master 127.0.99.190:34401 requested a full tablet report, sending...
I20240503 08:00:28.920569 15359 ts_manager.cc:194] Re-registered known tserver with Master: 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131:34211)
I20240503 08:00:28.921304 15359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:38169
I20240503 08:00:29.087309 15481 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.088019 15877 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.089342 16333 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:29.090304 16202 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:29.175200 15877 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.175242 16202 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:29.176239 16333 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:29.176851 15481 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.273662 16202 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:29.274854 15481 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.274950 15877 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.275305 16333 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:29.348839 15877 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:29.349609 16333 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:29.350306 16202 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:29.351686 15481 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:00:29.382936 15502 consensus_queue.cc:237] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } }
I20240503 08:00:29.384294 15899 raft_consensus.cc:1261] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:29.384616 16543 consensus_queue.cc:1035] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:29.386258 15966 raft_consensus.cc:2932] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0bfedb67806a457188db76fa09341dfd (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } } }
I20240503 08:00:29.386620 15899 raft_consensus.cc:2932] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0bfedb67806a457188db76fa09341dfd (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } } }
W20240503 08:00:29.387838 15436 consensus_peers.cc:485] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 -> Peer 0bfedb67806a457188db76fa09341dfd (127.0.99.130:38965): Couldn't send request to peer 0bfedb67806a457188db76fa09341dfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4b2d66e4b4147888ee7a3a953d24bee. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:29.389007 15356 catalog_manager.cc:5484] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index -1 to 4, NON_VOTER 0bfedb67806a457188db76fa09341dfd (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:29.395368 15899 consensus_queue.cc:237] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } }
I20240503 08:00:29.396679 15502 raft_consensus.cc:1261] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:29.397027 16556 consensus_queue.cc:1035] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:29.398523 15967 raft_consensus.cc:2932] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } } }
I20240503 08:00:29.399752 15502 raft_consensus.cc:2932] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } } }
W20240503 08:00:29.400625 15833 consensus_peers.cc:485] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 -> Peer 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131:34211): Couldn't send request to peer 6be8cb3537d84305a11b3a8bf04a9026. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d6c3ce2ccbc42fbbd66c578aa40e1f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:29.400516 15355 catalog_manager.cc:5484] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 reported cstate change: config changed from index -1 to 4, NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:29.406500 15502 consensus_queue.cc:237] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } }
I20240503 08:00:29.407588 15899 raft_consensus.cc:1261] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:29.407724 15436 consensus_peers.cc:485] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 -> Peer 0bfedb67806a457188db76fa09341dfd (127.0.99.130:38965): Couldn't send request to peer 0bfedb67806a457188db76fa09341dfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40cc8a45aece478bb35df85206a99b0f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:29.407910 16545 consensus_queue.cc:1035] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:29.409272 16543 raft_consensus.cc:2932] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0bfedb67806a457188db76fa09341dfd (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } } }
I20240503 08:00:29.409680 15899 raft_consensus.cc:2932] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0bfedb67806a457188db76fa09341dfd (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } } }
I20240503 08:00:29.411031 15356 catalog_manager.cc:5484] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index -1 to 4, NON_VOTER 0bfedb67806a457188db76fa09341dfd (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:29.417207 15899 consensus_queue.cc:237] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } }
I20240503 08:00:29.418198 15502 raft_consensus.cc:1261] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:29.418332 15833 consensus_peers.cc:485] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 -> Peer 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131:34211): Couldn't send request to peer 6be8cb3537d84305a11b3a8bf04a9026. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1f7a626b7a044d9918ff22e2f7bf583. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:29.418515 15967 consensus_queue.cc:1035] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:29.419817 16556 raft_consensus.cc:2932] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } } }
I20240503 08:00:29.420209 15502 raft_consensus.cc:2932] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } } }
I20240503 08:00:29.421787 15356 catalog_manager.cc:5484] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 reported cstate change: config changed from index -1 to 4, NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:29.427407 15502 consensus_queue.cc:237] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } }
I20240503 08:00:29.428349 15899 raft_consensus.cc:1261] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:00:29.428544 15436 consensus_peers.cc:485] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 -> Peer 0bfedb67806a457188db76fa09341dfd (127.0.99.130:38965): Couldn't send request to peer 0bfedb67806a457188db76fa09341dfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b83628c5f5f9459d88defa892f7a7fab. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:29.428658 16545 consensus_queue.cc:1035] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:29.430178 15966 raft_consensus.cc:2932] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0bfedb67806a457188db76fa09341dfd (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } } }
I20240503 08:00:29.430550 15899 raft_consensus.cc:2932] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0bfedb67806a457188db76fa09341dfd (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } } }
I20240503 08:00:29.432148 15356 catalog_manager.cc:5484] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index -1 to 4, NON_VOTER 0bfedb67806a457188db76fa09341dfd (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:29.437716 15502 consensus_queue.cc:237] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } }
I20240503 08:00:29.438663 15899 raft_consensus.cc:1261] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:29.438987 16545 consensus_queue.cc:1035] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:29.440333 16543 raft_consensus.cc:2932] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } } }
I20240503 08:00:29.440814 15899 raft_consensus.cc:2932] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } } }
W20240503 08:00:29.442394 15437 consensus_peers.cc:485] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 -> Peer 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131:34211): Couldn't send request to peer 6be8cb3537d84305a11b3a8bf04a9026. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9e4273eac3c484fa7e7af6bf0392702. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:29.442741 15356 catalog_manager.cc:5484] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index -1 to 4, NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:29.769973 16268 heartbeater.cc:503] Master 127.0.99.190:34401 was elected leader, sending a full tablet report...
I20240503 08:00:29.801092 16646 ts_tablet_manager.cc:902] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: Initiating tablet copy from peer fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:29.801687 16646 tablet_copy_client.cc:308] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35409
I20240503 08:00:29.805403 15523 tablet_copy_service.cc:140] P fec9c86608704e8e8fb88d3f6b6bf487: Received BeginTabletCopySession request for tablet b4b2d66e4b4147888ee7a3a953d24bee from peer 0bfedb67806a457188db76fa09341dfd ({username='slave'} at 127.0.99.130:34961)
I20240503 08:00:29.805531 15523 tablet_copy_service.cc:161] P fec9c86608704e8e8fb88d3f6b6bf487: Beginning new tablet copy session on tablet b4b2d66e4b4147888ee7a3a953d24bee from peer 0bfedb67806a457188db76fa09341dfd at {username='slave'} at 127.0.99.130:34961: session id = 0bfedb67806a457188db76fa09341dfd-b4b2d66e4b4147888ee7a3a953d24bee
I20240503 08:00:29.806623 15523 tablet_copy_source_session.cc:205] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:29.809494 16646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b2d66e4b4147888ee7a3a953d24bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:29.814150 16646 tablet_copy_client.cc:783] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:29.814316 16646 tablet_copy_client.cc:647] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:29.815690 16646 tablet_copy_client.cc:523] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:29.816952 16646 tablet_bootstrap.cc:492] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: Bootstrap starting.
I20240503 08:00:29.819049 16652 ts_tablet_manager.cc:902] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: Initiating tablet copy from peer 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:29.819664 16652 tablet_copy_client.cc:308] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37461
I20240503 08:00:29.822331 16646 log.cc:824] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:29.823243 15919 tablet_copy_service.cc:140] P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Received BeginTabletCopySession request for tablet f1f7a626b7a044d9918ff22e2f7bf583 from peer 6be8cb3537d84305a11b3a8bf04a9026 ({username='slave'} at 127.0.99.131:49807)
I20240503 08:00:29.823354 15919 tablet_copy_service.cc:161] P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Beginning new tablet copy session on tablet f1f7a626b7a044d9918ff22e2f7bf583 from peer 6be8cb3537d84305a11b3a8bf04a9026 at {username='slave'} at 127.0.99.131:49807: session id = 6be8cb3537d84305a11b3a8bf04a9026-f1f7a626b7a044d9918ff22e2f7bf583
I20240503 08:00:29.824308 15919 tablet_copy_source_session.cc:205] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:29.826880 16652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f7a626b7a044d9918ff22e2f7bf583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:29.831367 16652 tablet_copy_client.cc:783] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:29.831526 16652 tablet_copy_client.cc:647] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:29.832880 16652 tablet_copy_client.cc:523] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:29.835801 16652 tablet_bootstrap.cc:492] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: Bootstrap starting.
I20240503 08:00:29.840646 16652 log.cc:824] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:29.841212 16646 tablet_bootstrap.cc:492] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:29.841452 16646 tablet_bootstrap.cc:492] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: Bootstrap complete.
I20240503 08:00:29.841676 16646 ts_tablet_manager.cc:1367] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.003s
I20240503 08:00:29.843343 16646 raft_consensus.cc:348] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } }
I20240503 08:00:29.843524 16646 raft_consensus.cc:729] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0bfedb67806a457188db76fa09341dfd, State: Initialized, Role: LEARNER
I20240503 08:00:29.843791 16646 consensus_queue.cc:260] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } }
I20240503 08:00:29.844476 16646 ts_tablet_manager.cc:1398] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:00:29.845075 15523 tablet_copy_service.cc:342] P fec9c86608704e8e8fb88d3f6b6bf487: Request end of tablet copy session 0bfedb67806a457188db76fa09341dfd-b4b2d66e4b4147888ee7a3a953d24bee received from {username='slave'} at 127.0.99.130:34961
I20240503 08:00:29.845176 15523 tablet_copy_service.cc:433] P fec9c86608704e8e8fb88d3f6b6bf487: ending tablet copy session 0bfedb67806a457188db76fa09341dfd-b4b2d66e4b4147888ee7a3a953d24bee on tablet b4b2d66e4b4147888ee7a3a953d24bee with peer 0bfedb67806a457188db76fa09341dfd
I20240503 08:00:29.857713 16652 tablet_bootstrap.cc:492] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:29.857954 16652 tablet_bootstrap.cc:492] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: Bootstrap complete.
I20240503 08:00:29.858162 16652 ts_tablet_manager.cc:1367] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20240503 08:00:29.859776 16652 raft_consensus.cc:348] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } }
I20240503 08:00:29.859944 16652 raft_consensus.cc:729] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6be8cb3537d84305a11b3a8bf04a9026, State: Initialized, Role: LEARNER
I20240503 08:00:29.860213 16652 consensus_queue.cc:260] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } }
I20240503 08:00:29.860721 16399 heartbeater.cc:503] Master 127.0.99.190:34401 was elected leader, sending a full tablet report...
I20240503 08:00:29.860857 16652 ts_tablet_manager.cc:1398] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:00:29.861405 15919 tablet_copy_service.cc:342] P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Request end of tablet copy session 6be8cb3537d84305a11b3a8bf04a9026-f1f7a626b7a044d9918ff22e2f7bf583 received from {username='slave'} at 127.0.99.131:49807
I20240503 08:00:29.861495 15919 tablet_copy_service.cc:433] P 2703c9a1353f4e3c9f8bbb7b5e52eab5: ending tablet copy session 6be8cb3537d84305a11b3a8bf04a9026-f1f7a626b7a044d9918ff22e2f7bf583 on tablet f1f7a626b7a044d9918ff22e2f7bf583 with peer 6be8cb3537d84305a11b3a8bf04a9026
I20240503 08:00:29.914824 16646 ts_tablet_manager.cc:902] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: Initiating tablet copy from peer fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:29.915227 16646 tablet_copy_client.cc:308] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35409
I20240503 08:00:29.915629 15523 tablet_copy_service.cc:140] P fec9c86608704e8e8fb88d3f6b6bf487: Received BeginTabletCopySession request for tablet b83628c5f5f9459d88defa892f7a7fab from peer 0bfedb67806a457188db76fa09341dfd ({username='slave'} at 127.0.99.130:34961)
I20240503 08:00:29.915724 15523 tablet_copy_service.cc:161] P fec9c86608704e8e8fb88d3f6b6bf487: Beginning new tablet copy session on tablet b83628c5f5f9459d88defa892f7a7fab from peer 0bfedb67806a457188db76fa09341dfd at {username='slave'} at 127.0.99.130:34961: session id = 0bfedb67806a457188db76fa09341dfd-b83628c5f5f9459d88defa892f7a7fab
I20240503 08:00:29.916532 15523 tablet_copy_source_session.cc:205] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:29.917553 16652 ts_tablet_manager.cc:902] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: Initiating tablet copy from peer 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461)
I20240503 08:00:29.917655 16646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b83628c5f5f9459d88defa892f7a7fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:29.917914 16652 tablet_copy_client.cc:308] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:37461
I20240503 08:00:29.918390 15919 tablet_copy_service.cc:140] P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Received BeginTabletCopySession request for tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4 from peer 6be8cb3537d84305a11b3a8bf04a9026 ({username='slave'} at 127.0.99.131:49807)
I20240503 08:00:29.918493 15919 tablet_copy_service.cc:161] P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Beginning new tablet copy session on tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4 from peer 6be8cb3537d84305a11b3a8bf04a9026 at {username='slave'} at 127.0.99.131:49807: session id = 6be8cb3537d84305a11b3a8bf04a9026-9d6c3ce2ccbc42fbbd66c578aa40e1f4
I20240503 08:00:29.919312 15919 tablet_copy_source_session.cc:205] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:29.919531 16646 tablet_copy_client.cc:783] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:29.919642 16646 tablet_copy_client.cc:647] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:29.920086 16652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:29.921929 16652 tablet_copy_client.cc:783] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:29.922019 16652 tablet_copy_client.cc:647] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:29.923156 16652 tablet_copy_client.cc:523] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:29.923291 16646 tablet_copy_client.cc:523] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:29.924217 16652 tablet_bootstrap.cc:492] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: Bootstrap starting.
I20240503 08:00:29.924278 16646 tablet_bootstrap.cc:492] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: Bootstrap starting.
I20240503 08:00:29.946841 16646 tablet_bootstrap.cc:492] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:29.947088 16646 tablet_bootstrap.cc:492] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: Bootstrap complete.
I20240503 08:00:29.947247 16646 ts_tablet_manager.cc:1367] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20240503 08:00:29.947767 16646 raft_consensus.cc:348] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } }
I20240503 08:00:29.947887 16646 raft_consensus.cc:729] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0bfedb67806a457188db76fa09341dfd, State: Initialized, Role: LEARNER
I20240503 08:00:29.947988 16646 consensus_queue.cc:260] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } }
I20240503 08:00:29.948241 16646 ts_tablet_manager.cc:1398] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:29.948784 15523 tablet_copy_service.cc:342] P fec9c86608704e8e8fb88d3f6b6bf487: Request end of tablet copy session 0bfedb67806a457188db76fa09341dfd-b83628c5f5f9459d88defa892f7a7fab received from {username='slave'} at 127.0.99.130:34961
I20240503 08:00:29.948863 15523 tablet_copy_service.cc:433] P fec9c86608704e8e8fb88d3f6b6bf487: ending tablet copy session 0bfedb67806a457188db76fa09341dfd-b83628c5f5f9459d88defa892f7a7fab on tablet b83628c5f5f9459d88defa892f7a7fab with peer 0bfedb67806a457188db76fa09341dfd
I20240503 08:00:29.955925 16652 tablet_bootstrap.cc:492] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:29.956104 16652 tablet_bootstrap.cc:492] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: Bootstrap complete.
I20240503 08:00:29.956215 16652 ts_tablet_manager.cc:1367] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.000s
I20240503 08:00:29.956596 16652 raft_consensus.cc:348] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } }
I20240503 08:00:29.956684 16652 raft_consensus.cc:729] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6be8cb3537d84305a11b3a8bf04a9026, State: Initialized, Role: LEARNER
I20240503 08:00:29.956756 16652 consensus_queue.cc:260] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } }
I20240503 08:00:29.956969 16652 ts_tablet_manager.cc:1398] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:29.957376 15919 tablet_copy_service.cc:342] P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Request end of tablet copy session 6be8cb3537d84305a11b3a8bf04a9026-9d6c3ce2ccbc42fbbd66c578aa40e1f4 received from {username='slave'} at 127.0.99.131:49807
I20240503 08:00:29.957440 15919 tablet_copy_service.cc:433] P 2703c9a1353f4e3c9f8bbb7b5e52eab5: ending tablet copy session 6be8cb3537d84305a11b3a8bf04a9026-9d6c3ce2ccbc42fbbd66c578aa40e1f4 on tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4 with peer 6be8cb3537d84305a11b3a8bf04a9026
I20240503 08:00:29.966055 16652 ts_tablet_manager.cc:902] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: Initiating tablet copy from peer fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:29.966405 16652 tablet_copy_client.cc:308] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35409
I20240503 08:00:29.969347 15523 tablet_copy_service.cc:140] P fec9c86608704e8e8fb88d3f6b6bf487: Received BeginTabletCopySession request for tablet c9e4273eac3c484fa7e7af6bf0392702 from peer 6be8cb3537d84305a11b3a8bf04a9026 ({username='slave'} at 127.0.99.131:48877)
I20240503 08:00:29.969430 15523 tablet_copy_service.cc:161] P fec9c86608704e8e8fb88d3f6b6bf487: Beginning new tablet copy session on tablet c9e4273eac3c484fa7e7af6bf0392702 from peer 6be8cb3537d84305a11b3a8bf04a9026 at {username='slave'} at 127.0.99.131:48877: session id = 6be8cb3537d84305a11b3a8bf04a9026-c9e4273eac3c484fa7e7af6bf0392702
I20240503 08:00:29.970039 15523 tablet_copy_source_session.cc:205] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:29.970670 16652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4273eac3c484fa7e7af6bf0392702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:29.972215 16652 tablet_copy_client.cc:783] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:29.972333 16652 tablet_copy_client.cc:647] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:29.973467 16652 tablet_copy_client.cc:523] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:29.974326 16652 tablet_bootstrap.cc:492] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: Bootstrap starting.
I20240503 08:00:29.989243 16652 tablet_bootstrap.cc:492] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:29.989437 16652 tablet_bootstrap.cc:492] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: Bootstrap complete.
I20240503 08:00:29.989563 16652 ts_tablet_manager.cc:1367] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: Time spent bootstrapping tablet: real 0.015s user 0.015s sys 0.000s
I20240503 08:00:29.989902 16652 raft_consensus.cc:348] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } }
I20240503 08:00:29.989989 16652 raft_consensus.cc:729] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6be8cb3537d84305a11b3a8bf04a9026, State: Initialized, Role: LEARNER
I20240503 08:00:29.990073 16652 consensus_queue.cc:260] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: true } }
I20240503 08:00:29.990288 16652 ts_tablet_manager.cc:1398] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:29.990738 15523 tablet_copy_service.cc:342] P fec9c86608704e8e8fb88d3f6b6bf487: Request end of tablet copy session 6be8cb3537d84305a11b3a8bf04a9026-c9e4273eac3c484fa7e7af6bf0392702 received from {username='slave'} at 127.0.99.131:48877
I20240503 08:00:29.990841 15523 tablet_copy_service.cc:433] P fec9c86608704e8e8fb88d3f6b6bf487: ending tablet copy session 6be8cb3537d84305a11b3a8bf04a9026-c9e4273eac3c484fa7e7af6bf0392702 on tablet c9e4273eac3c484fa7e7af6bf0392702 with peer 6be8cb3537d84305a11b3a8bf04a9026
I20240503 08:00:30.032017 16646 ts_tablet_manager.cc:902] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: Initiating tablet copy from peer fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:30.032471 16646 tablet_copy_client.cc:308] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35409
I20240503 08:00:30.032960 15523 tablet_copy_service.cc:140] P fec9c86608704e8e8fb88d3f6b6bf487: Received BeginTabletCopySession request for tablet 40cc8a45aece478bb35df85206a99b0f from peer 0bfedb67806a457188db76fa09341dfd ({username='slave'} at 127.0.99.130:34961)
I20240503 08:00:30.033089 15523 tablet_copy_service.cc:161] P fec9c86608704e8e8fb88d3f6b6bf487: Beginning new tablet copy session on tablet 40cc8a45aece478bb35df85206a99b0f from peer 0bfedb67806a457188db76fa09341dfd at {username='slave'} at 127.0.99.130:34961: session id = 0bfedb67806a457188db76fa09341dfd-40cc8a45aece478bb35df85206a99b0f
I20240503 08:00:30.033944 15523 tablet_copy_source_session.cc:205] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:30.034811 16646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40cc8a45aece478bb35df85206a99b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:30.036759 16646 tablet_copy_client.cc:783] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:30.036898 16646 tablet_copy_client.cc:647] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:30.037923 16646 tablet_copy_client.cc:523] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:30.038803 16646 tablet_bootstrap.cc:492] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: Bootstrap starting.
I20240503 08:00:30.055044 16646 tablet_bootstrap.cc:492] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:30.055305 16646 tablet_bootstrap.cc:492] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: Bootstrap complete.
I20240503 08:00:30.055454 16646 ts_tablet_manager.cc:1367] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20240503 08:00:30.055795 16646 raft_consensus.cc:348] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } }
I20240503 08:00:30.055892 16646 raft_consensus.cc:729] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0bfedb67806a457188db76fa09341dfd, State: Initialized, Role: LEARNER
I20240503 08:00:30.055977 16646 consensus_queue.cc:260] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: true } }
I20240503 08:00:30.056187 16646 ts_tablet_manager.cc:1398] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:30.056629 15523 tablet_copy_service.cc:342] P fec9c86608704e8e8fb88d3f6b6bf487: Request end of tablet copy session 0bfedb67806a457188db76fa09341dfd-40cc8a45aece478bb35df85206a99b0f received from {username='slave'} at 127.0.99.130:34961
I20240503 08:00:30.056694 15523 tablet_copy_service.cc:433] P fec9c86608704e8e8fb88d3f6b6bf487: ending tablet copy session 0bfedb67806a457188db76fa09341dfd-40cc8a45aece478bb35df85206a99b0f on tablet 40cc8a45aece478bb35df85206a99b0f with peer 0bfedb67806a457188db76fa09341dfd
I20240503 08:00:30.313213 16222 raft_consensus.cc:1203] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:30.391270 16353 raft_consensus.cc:1203] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:30.418406 16353 raft_consensus.cc:1203] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:30.467698 16353 raft_consensus.cc:1203] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:30.519778 16222 raft_consensus.cc:1203] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:30.586585 16222 raft_consensus.cc:1203] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:00:30.743691 16742 raft_consensus.cc:1050] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487: attempting to promote NON_VOTER 0bfedb67806a457188db76fa09341dfd to VOTER
I20240503 08:00:30.744138 16742 consensus_queue.cc:237] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } }
I20240503 08:00:30.745430 15899 raft_consensus.cc:1261] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:30.745453 16222 raft_consensus.cc:1261] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:30.745797 15966 consensus_queue.cc:1035] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:30.745992 16743 consensus_queue.cc:1035] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:30.747656 15966 raft_consensus.cc:2932] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.748096 15899 raft_consensus.cc:2932] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.748999 16222 raft_consensus.cc:2932] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.750138 15356 catalog_manager.cc:5484] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:30.753217 15502 consensus_queue.cc:237] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } }
I20240503 08:00:30.754074 16222 raft_consensus.cc:1261] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:30.754390 16742 consensus_queue.cc:1035] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:30.755640 16743 raft_consensus.cc:2932] T b4b2d66e4b4147888ee7a3a953d24bee P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.756031 16222 raft_consensus.cc:2932] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.757172 15346 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b4b2d66e4b4147888ee7a3a953d24bee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:30.757906 15355 catalog_manager.cc:5484] T b4b2d66e4b4147888ee7a3a953d24bee P 0bfedb67806a457188db76fa09341dfd reported cstate change: config changed from index 5 to 6, VOTER 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.759924 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b2d66e4b4147888ee7a3a953d24bee with delete_type TABLET_DATA_TOMBSTONED (TS 2703c9a1353f4e3c9f8bbb7b5e52eab5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57384
I20240503 08:00:30.760380 16813 tablet_replica.cc:330] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5: stopping tablet replica
I20240503 08:00:30.760609 16813 raft_consensus.cc:2229] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:30.760757 16813 raft_consensus.cc:2258] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:30.761695 16813 ts_tablet_manager.cc:1875] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:30.763372 16813 ts_tablet_manager.cc:1888] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:30.763432 16813 log.cc:1196] T b4b2d66e4b4147888ee7a3a953d24bee P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b4b2d66e4b4147888ee7a3a953d24bee
I20240503 08:00:30.764003 15347 catalog_manager.cc:4830] TS 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461): tablet b4b2d66e4b4147888ee7a3a953d24bee (table rebalance_test_table_2 [id=050c272c47d64b5498447aaf07125871]) successfully deleted
I20240503 08:00:30.922667 16742 raft_consensus.cc:1050] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487: attempting to promote NON_VOTER 0bfedb67806a457188db76fa09341dfd to VOTER
I20240503 08:00:30.923007 16742 consensus_queue.cc:237] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } }
I20240503 08:00:30.924043 16222 raft_consensus.cc:1261] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:30.924135 15899 raft_consensus.cc:1261] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:30.924391 16743 consensus_queue.cc:1035] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:30.924571 16742 consensus_queue.cc:1035] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:30.926234 15966 raft_consensus.cc:2932] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.926630 15899 raft_consensus.cc:2932] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.927585 16222 raft_consensus.cc:2932] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.928743 15356 catalog_manager.cc:5484] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:30.931505 15502 consensus_queue.cc:237] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } }
I20240503 08:00:30.932322 16222 raft_consensus.cc:1261] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:30.932627 15966 consensus_queue.cc:1035] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:30.933900 16742 raft_consensus.cc:2932] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.934273 16222 raft_consensus.cc:2932] T b83628c5f5f9459d88defa892f7a7fab P 0bfedb67806a457188db76fa09341dfd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:30.935600 15346 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b83628c5f5f9459d88defa892f7a7fab with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:30.936146 15356 catalog_manager.cc:5484] T b83628c5f5f9459d88defa892f7a7fab P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index 5 to 6, VOTER 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:30.938174 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet b83628c5f5f9459d88defa892f7a7fab with delete_type TABLET_DATA_TOMBSTONED (TS 2703c9a1353f4e3c9f8bbb7b5e52eab5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57384
I20240503 08:00:30.938381 16813 tablet_replica.cc:330] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5: stopping tablet replica
I20240503 08:00:30.938488 16813 raft_consensus.cc:2229] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:30.938620 16813 raft_consensus.cc:2258] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:30.939201 16813 ts_tablet_manager.cc:1875] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:30.940742 16813 ts_tablet_manager.cc:1888] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:30.940801 16813 log.cc:1196] T b83628c5f5f9459d88defa892f7a7fab P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b83628c5f5f9459d88defa892f7a7fab
I20240503 08:00:30.941233 15347 catalog_manager.cc:4830] TS 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461): tablet b83628c5f5f9459d88defa892f7a7fab (table rebalance_test_table_0 [id=a86ef2e7fcf8450fa5f40ab84054bf6d]) successfully deleted
I20240503 08:00:31.002888 16775 raft_consensus.cc:1050] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: attempting to promote NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 to VOTER
I20240503 08:00:31.002956 16872 raft_consensus.cc:1050] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5: attempting to promote NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 to VOTER
I20240503 08:00:31.003334 16872 consensus_queue.cc:237] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } }
I20240503 08:00:31.003335 16775 consensus_queue.cc:237] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } }
I20240503 08:00:31.004627 15502 raft_consensus.cc:1261] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:31.004684 15503 raft_consensus.cc:1261] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:31.004748 16352 raft_consensus.cc:1261] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:31.005129 16876 consensus_queue.cc:1035] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:31.005142 16878 consensus_queue.cc:1035] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:31.005455 16873 consensus_queue.cc:1035] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:31.005488 16352 raft_consensus.cc:1261] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:31.005774 16873 consensus_queue.cc:1035] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:31.007459 16878 raft_consensus.cc:2932] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.007833 15502 raft_consensus.cc:2932] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.008651 16775 raft_consensus.cc:2932] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.010040 15356 catalog_manager.cc:5484] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.010502 15502 raft_consensus.cc:2932] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.011029 16352 raft_consensus.cc:2932] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.012531 16353 raft_consensus.cc:2932] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.013301 15359 catalog_manager.cc:5484] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026 reported cstate change: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.016142 15899 consensus_queue.cc:237] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } }
I20240503 08:00:31.016166 15898 consensus_queue.cc:237] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } }
I20240503 08:00:31.016957 16353 raft_consensus.cc:1261] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 FOLLOWER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:31.017087 16352 raft_consensus.cc:1261] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 FOLLOWER]: Refusing update from remote peer 2703c9a1353f4e3c9f8bbb7b5e52eab5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:31.017295 16878 consensus_queue.cc:1035] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:31.017364 16873 consensus_queue.cc:1035] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:31.018600 16877 raft_consensus.cc:2932] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.018617 16872 raft_consensus.cc:2932] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.018970 16353 raft_consensus.cc:2932] T f1f7a626b7a044d9918ff22e2f7bf583 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.020532 15356 catalog_manager.cc:5484] T f1f7a626b7a044d9918ff22e2f7bf583 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 reported cstate change: config changed from index 5 to 6, VOTER fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:31.020730 15966 raft_consensus.cc:1050] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: attempting to promote NON_VOTER 6be8cb3537d84305a11b3a8bf04a9026 to VOTER
I20240503 08:00:31.020707 16352 raft_consensus.cc:2932] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.021050 15966 consensus_queue.cc:237] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } }
I20240503 08:00:31.022311 15899 raft_consensus.cc:1261] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:31.022657 15966 consensus_queue.cc:1035] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:31.025503 16352 raft_consensus.cc:1261] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 LEARNER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:31.025858 16742 consensus_queue.cc:1035] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:31.025976 15347 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1f7a626b7a044d9918ff22e2f7bf583 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:31.026101 15347 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:31.026238 15481 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f7a626b7a044d9918ff22e2f7bf583 with delete_type TABLET_DATA_TOMBSTONED (TS fec9c86608704e8e8fb88d3f6b6bf487 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52254
I20240503 08:00:31.026858 16742 raft_consensus.cc:2932] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.027292 15355 catalog_manager.cc:5484] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 reported cstate change: config changed from index 5 to 6, VOTER fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:31.028813 15899 raft_consensus.cc:2932] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.029314 15359 catalog_manager.cc:5484] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:31.029597 15481 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4 with delete_type TABLET_DATA_TOMBSTONED (TS fec9c86608704e8e8fb88d3f6b6bf487 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52254
I20240503 08:00:31.033262 16886 tablet_replica.cc:330] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487: stopping tablet replica
I20240503 08:00:31.033521 16886 raft_consensus.cc:2229] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:31.033697 16886 raft_consensus.cc:2258] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:31.034798 16886 ts_tablet_manager.cc:1875] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:31.036540 16886 ts_tablet_manager.cc:1888] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:31.036630 16886 log.cc:1196] T f1f7a626b7a044d9918ff22e2f7bf583 P fec9c86608704e8e8fb88d3f6b6bf487: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f1f7a626b7a044d9918ff22e2f7bf583
I20240503 08:00:31.036718 16352 raft_consensus.cc:2932] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.037101 16886 tablet_replica.cc:330] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487: stopping tablet replica
I20240503 08:00:31.037108 15346 catalog_manager.cc:4830] TS fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409): tablet f1f7a626b7a044d9918ff22e2f7bf583 (table rebalance_test_table_1 [id=75b65d480fb4494faa4a430e500f8f2e]) successfully deleted
I20240503 08:00:31.037220 16886 raft_consensus.cc:2229] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:31.037349 16886 raft_consensus.cc:2258] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:31.037865 16886 ts_tablet_manager.cc:1875] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:31.039250 16886 ts_tablet_manager.cc:1888] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:31.039309 16886 log.cc:1196] T 9d6c3ce2ccbc42fbbd66c578aa40e1f4 P fec9c86608704e8e8fb88d3f6b6bf487: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9d6c3ce2ccbc42fbbd66c578aa40e1f4
I20240503 08:00:31.039678 15346 catalog_manager.cc:4830] TS fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409): tablet 9d6c3ce2ccbc42fbbd66c578aa40e1f4 (table rebalance_test_table_0 [id=a86ef2e7fcf8450fa5f40ab84054bf6d]) successfully deleted
I20240503 08:00:31.086719 15502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9e4273eac3c484fa7e7af6bf0392702"
dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52730
I20240503 08:00:31.086877 15502 raft_consensus.cc:595] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:31.111811 15502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9e4273eac3c484fa7e7af6bf0392702"
dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52756
I20240503 08:00:31.111935 15502 raft_consensus.cc:595] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:31.199110 15966 raft_consensus.cc:1050] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487: attempting to promote NON_VOTER 0bfedb67806a457188db76fa09341dfd to VOTER
I20240503 08:00:31.199479 15966 consensus_queue.cc:237] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } }
I20240503 08:00:31.200691 16222 raft_consensus.cc:1261] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd [term 1 LEARNER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:31.200722 15899 raft_consensus.cc:1261] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:31.201042 16742 consensus_queue.cc:1035] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:31.201254 15966 consensus_queue.cc:1035] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:31.202862 16742 raft_consensus.cc:2932] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:31.203307 15899 raft_consensus.cc:2932] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:31.203887 16222 raft_consensus.cc:2932] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:31.205706 15359 catalog_manager.cc:5484] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index 4 to 5, 0bfedb67806a457188db76fa09341dfd (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:31.208039 15502 consensus_queue.cc:237] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } }
I20240503 08:00:31.208840 16222 raft_consensus.cc:1261] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd [term 1 FOLLOWER]: Refusing update from remote peer fec9c86608704e8e8fb88d3f6b6bf487: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:31.209172 15966 consensus_queue.cc:1035] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:31.210376 16743 raft_consensus.cc:2932] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:31.210743 16222 raft_consensus.cc:2932] T 40cc8a45aece478bb35df85206a99b0f P 0bfedb67806a457188db76fa09341dfd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } } }
I20240503 08:00:31.211691 15346 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40cc8a45aece478bb35df85206a99b0f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:31.212363 15355 catalog_manager.cc:5484] T 40cc8a45aece478bb35df85206a99b0f P fec9c86608704e8e8fb88d3f6b6bf487 reported cstate change: config changed from index 5 to 6, VOTER 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfedb67806a457188db76fa09341dfd" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:31.214360 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet 40cc8a45aece478bb35df85206a99b0f with delete_type TABLET_DATA_TOMBSTONED (TS 2703c9a1353f4e3c9f8bbb7b5e52eab5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57384
I20240503 08:00:31.214579 16813 tablet_replica.cc:330] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5: stopping tablet replica
I20240503 08:00:31.214702 16813 raft_consensus.cc:2229] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:31.214839 16813 raft_consensus.cc:2258] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:31.215389 16813 ts_tablet_manager.cc:1875] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:31.216997 16813 ts_tablet_manager.cc:1888] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:00:31.217082 16813 log.cc:1196] T 40cc8a45aece478bb35df85206a99b0f P 2703c9a1353f4e3c9f8bbb7b5e52eab5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/40cc8a45aece478bb35df85206a99b0f
I20240503 08:00:31.217483 15347 catalog_manager.cc:4830] TS 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461): tablet 40cc8a45aece478bb35df85206a99b0f (table rebalance_test_table_1 [id=75b65d480fb4494faa4a430e500f8f2e]) successfully deleted
I20240503 08:00:31.326037 15502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9e4273eac3c484fa7e7af6bf0392702"
dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52782
I20240503 08:00:31.326167 15502 raft_consensus.cc:595] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:31.339363 15502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9e4273eac3c484fa7e7af6bf0392702"
dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52810
I20240503 08:00:31.339480 15502 raft_consensus.cc:595] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:00:31.518843 15966 raft_consensus.cc:979] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: : Instructing follower 6be8cb3537d84305a11b3a8bf04a9026 to start an election
I20240503 08:00:31.518977 15966 raft_consensus.cc:1067] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Signalling peer 6be8cb3537d84305a11b3a8bf04a9026 to start an election
I20240503 08:00:31.519562 16352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e4273eac3c484fa7e7af6bf0392702"
dest_uuid: "6be8cb3537d84305a11b3a8bf04a9026"
from {username='slave'} at 127.0.99.129:57265
I20240503 08:00:31.519752 16352 raft_consensus.cc:482] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:00:31.519855 16352 raft_consensus.cc:3037] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:31.521438 16352 raft_consensus.cc:504] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } }
I20240503 08:00:31.522423 16352 leader_election.cc:288] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [CANDIDATE]: Term 2 election: Requested vote from peers 2703c9a1353f4e3c9f8bbb7b5e52eab5 (127.0.99.132:37461), fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409)
I20240503 08:00:31.522738 15899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4273eac3c484fa7e7af6bf0392702" candidate_uuid: "6be8cb3537d84305a11b3a8bf04a9026" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5"
I20240503 08:00:31.522873 15899 raft_consensus.cc:3037] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:31.522842 15502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4273eac3c484fa7e7af6bf0392702" candidate_uuid: "6be8cb3537d84305a11b3a8bf04a9026" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fec9c86608704e8e8fb88d3f6b6bf487"
I20240503 08:00:31.522953 15502 raft_consensus.cc:3032] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:00:31.523001 15502 raft_consensus.cc:729] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fec9c86608704e8e8fb88d3f6b6bf487, State: Running, Role: LEADER
I20240503 08:00:31.523176 15502 consensus_queue.cc:260] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } }
I20240503 08:00:31.523316 15502 raft_consensus.cc:3037] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:00:31.523715 15899 raft_consensus.cc:2454] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be8cb3537d84305a11b3a8bf04a9026 in term 2.
I20240503 08:00:31.524034 15502 raft_consensus.cc:2454] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6be8cb3537d84305a11b3a8bf04a9026 in term 2.
I20240503 08:00:31.524118 16288 leader_election.cc:302] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703c9a1353f4e3c9f8bbb7b5e52eab5, 6be8cb3537d84305a11b3a8bf04a9026; no voters:
I20240503 08:00:31.524358 16883 raft_consensus.cc:2781] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:00:31.524570 16883 raft_consensus.cc:686] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 2 LEADER]: Becoming Leader. State: Replica: 6be8cb3537d84305a11b3a8bf04a9026, State: Running, Role: LEADER
I20240503 08:00:31.524813 16883 consensus_queue.cc:237] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } }
I20240503 08:00:31.526660 15356 catalog_manager.cc:5484] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 reported cstate change: term changed from 1 to 2, leader changed from fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) to 6be8cb3537d84305a11b3a8bf04a9026 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "6be8cb3537d84305a11b3a8bf04a9026" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:31.938685 15899 raft_consensus.cc:1261] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 2 FOLLOWER]: Refusing update from remote peer 6be8cb3537d84305a11b3a8bf04a9026: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:31.939199 16883 consensus_queue.cc:1035] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:31.941573 15502 raft_consensus.cc:1261] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 2 FOLLOWER]: Refusing update from remote peer 6be8cb3537d84305a11b3a8bf04a9026: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:00:31.942298 16883 consensus_queue.cc:1035] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec9c86608704e8e8fb88d3f6b6bf487" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:31.946806 16352 consensus_queue.cc:237] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } }
I20240503 08:00:31.947624 15899 raft_consensus.cc:1261] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 2 FOLLOWER]: Refusing update from remote peer 6be8cb3537d84305a11b3a8bf04a9026: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:00:31.947933 16883 consensus_queue.cc:1035] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:31.949200 16961 raft_consensus.cc:2932] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.949688 15899 raft_consensus.cc:2932] T c9e4273eac3c484fa7e7af6bf0392702 P 2703c9a1353f4e3c9f8bbb7b5e52eab5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } } }
I20240503 08:00:31.951017 15346 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9e4273eac3c484fa7e7af6bf0392702 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:31.951380 15356 catalog_manager.cc:5484] T c9e4273eac3c484fa7e7af6bf0392702 P 6be8cb3537d84305a11b3a8bf04a9026 reported cstate change: config changed from index 5 to 7, VOTER fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "6be8cb3537d84305a11b3a8bf04a9026" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2703c9a1353f4e3c9f8bbb7b5e52eab5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6be8cb3537d84305a11b3a8bf04a9026" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:31.953567 15481 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e4273eac3c484fa7e7af6bf0392702 with delete_type TABLET_DATA_TOMBSTONED (TS fec9c86608704e8e8fb88d3f6b6bf487 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52254
I20240503 08:00:31.953853 16981 tablet_replica.cc:330] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: stopping tablet replica
I20240503 08:00:31.953984 16981 raft_consensus.cc:2229] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:31.954080 16981 raft_consensus.cc:2258] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:31.954639 16981 ts_tablet_manager.cc:1875] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:31.956526 16981 ts_tablet_manager.cc:1888] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:00:31.956601 16981 log.cc:1196] T c9e4273eac3c484fa7e7af6bf0392702 P fec9c86608704e8e8fb88d3f6b6bf487: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c9e4273eac3c484fa7e7af6bf0392702
I20240503 08:00:31.957026 15346 catalog_manager.cc:4830] TS fec9c86608704e8e8fb88d3f6b6bf487 (127.0.99.129:35409): tablet c9e4273eac3c484fa7e7af6bf0392702 (table rebalance_test_table_2 [id=050c272c47d64b5498447aaf07125871]) successfully deleted
I20240503 08:00:32.002388 15877 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.003347 16333 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.003600 16202 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:32.015101 15481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:32.088371 16333 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.088364 16202 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:32.091178 15481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:32.092186 15877 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.158049 16202 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:32.160051 15481 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:32.161231 15877 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:32.162740 16333 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:32.193437 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 15421
I20240503 08:00:32.199100 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 16140
I20240503 08:00:32.203562 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 16271
I20240503 08:00:32.207929 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 15816
I20240503 08:00:32.212577 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 15330
2024-05-03T08:00:32Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (10162 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx6_rf7_t7_l_3_4_3
I20240503 08:00:32.234910 398 test_util.cc:268] Using random seed: 488979866
I20240503 08:00:32.235754 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:00:32.235803 398 ts_itest-base.cc:113] --------------
I20240503 08:00:32.235819 398 ts_itest-base.cc:114] 10 tablet servers
I20240503 08:00:32.235832 398 ts_itest-base.cc:115] 7 replicas per TS
I20240503 08:00:32.235858 398 ts_itest-base.cc:116] --------------
2024-05-03T08:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:00:32Z Disabled control of system clock
I20240503 08:00:32.244503 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35233
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:35233
--location_mapping_cmd=/tmp/dist-test-taskD2FJ4a/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 08:00:32.341749 17093 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:00:32.341948 17093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:32.341970 17093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:32.342010 17093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:32.344825 17093 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:32.344878 17093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:00:32.344904 17093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:32.344928 17093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:00:32.344942 17093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:00:32.348336 17093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskD2FJ4a/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:35233
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.17093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:32.349026 17093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:32.349998 17093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:32.357020 17098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.357573 17099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.358495 17101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.359184 17093 server_base.cc:1034] running on GCE node
I20240503 08:00:32.359596 17093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:32.360364 17093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:32.361491 17093 hybrid_clock.cc:648] HybridClock initialized: now 1714723232361475 us; error 32 us; skew 500 ppm
I20240503 08:00:32.366252 17093 webserver.cc:460] Webserver started at http://127.0.99.190:44285/ using document root <none> and password file <none>
I20240503 08:00:32.366744 17093 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:32.366794 17093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:32.366957 17093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:32.368474 17093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "51daab6ec19c401bbaa500647c2d47dd"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.368892 17093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "51daab6ec19c401bbaa500647c2d47dd"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.371829 17093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:32.373752 17107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.374639 17093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:32.374735 17093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "51daab6ec19c401bbaa500647c2d47dd"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.374806 17093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:32.385290 17093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:32.385880 17093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:32.386042 17093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:32.391819 17093 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:35233
I20240503 08:00:32.391819 17159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:35233 every 8 connection(s)
I20240503 08:00:32.392686 17093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:00:32.395047 17160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:32.398294 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 17093
I20240503 08:00:32.398415 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:00:32.400161 17160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd: Bootstrap starting.
I20240503 08:00:32.402058 17160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:32.402681 17160 log.cc:824] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:32.404110 17160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd: No bootstrap required, opened a new log
I20240503 08:00:32.406713 17160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51daab6ec19c401bbaa500647c2d47dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35233 } }
I20240503 08:00:32.406912 17160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:32.406976 17160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51daab6ec19c401bbaa500647c2d47dd, State: Initialized, Role: FOLLOWER
I20240503 08:00:32.407338 17160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51daab6ec19c401bbaa500647c2d47dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35233 } }
I20240503 08:00:32.407451 17160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:00:32.407505 17160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:00:32.407596 17160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:32.408236 17160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51daab6ec19c401bbaa500647c2d47dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35233 } }
I20240503 08:00:32.408533 17160 leader_election.cc:302] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51daab6ec19c401bbaa500647c2d47dd; no voters:
I20240503 08:00:32.408748 17160 leader_election.cc:288] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:00:32.408849 17165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:32.409083 17165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [term 1 LEADER]: Becoming Leader. State: Replica: 51daab6ec19c401bbaa500647c2d47dd, State: Running, Role: LEADER
I20240503 08:00:32.409384 17165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51daab6ec19c401bbaa500647c2d47dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35233 } }
I20240503 08:00:32.409724 17160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:00:32.410748 17167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51daab6ec19c401bbaa500647c2d47dd. Latest consensus state: current_term: 1 leader_uuid: "51daab6ec19c401bbaa500647c2d47dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51daab6ec19c401bbaa500647c2d47dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35233 } } }
I20240503 08:00:32.410748 17166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51daab6ec19c401bbaa500647c2d47dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51daab6ec19c401bbaa500647c2d47dd" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35233 } } }
I20240503 08:00:32.410895 17167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:32.410895 17166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd [sys.catalog]: This master's current role is: LEADER
I20240503 08:00:32.411243 17174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:00:32.413574 17174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:00:32.418061 17174 catalog_manager.cc:1347] Generated new cluster ID: 1f76f51cd85f47a4830e37dda628b724
I20240503 08:00:32.418120 17174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:00:32.433514 17174 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:00:32.434298 17174 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:00:32.441188 17174 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 51daab6ec19c401bbaa500647c2d47dd: Generated new TSK 0
I20240503 08:00:32.441763 17174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:00:32.451746 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:00:32.553000 17184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:32.553232 17184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:32.553277 17184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:32.556211 17184 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:32.556263 17184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:32.556337 17184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 08:00:32.559762 17184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.17184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:32.560520 17184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:32.561548 17184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:32.569489 17189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.570019 17192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.570951 17190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.571663 17184 server_base.cc:1034] running on GCE node
I20240503 08:00:32.571987 17184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:32.572466 17184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:32.573599 17184 hybrid_clock.cc:648] HybridClock initialized: now 1714723232573585 us; error 26 us; skew 500 ppm
I20240503 08:00:32.575245 17184 webserver.cc:460] Webserver started at http://127.0.99.129:36679/ using document root <none> and password file <none>
I20240503 08:00:32.575726 17184 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:32.575775 17184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:32.575944 17184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:32.577507 17184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8eb6822290b046f88c861159aa9421ec"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.577916 17184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8eb6822290b046f88c861159aa9421ec"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.580797 17184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:00:32.582736 17198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.583650 17184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:32.583734 17184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8eb6822290b046f88c861159aa9421ec"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.583796 17184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:32.598862 17184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:32.599298 17184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:32.599409 17184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:32.599809 17184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:32.600550 17184 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:32.600592 17184 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.600626 17184 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:32.600646 17184 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.607913 17184 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33199
I20240503 08:00:32.607939 17311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33199 every 8 connection(s)
I20240503 08:00:32.608676 17184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:00:32.616209 17312 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:32.616389 17312 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:32.616629 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 17184
I20240503 08:00:32.616721 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:00:32.616864 17312 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:32.618515 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:32.651126 17123 ts_manager.cc:194] Registered new tserver with Master: 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:32.652460 17123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:40227
W20240503 08:00:32.719039 17316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:32.719230 17316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:32.719274 17316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:32.722272 17316 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:32.722342 17316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:32.722436 17316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 08:00:32.725890 17316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.17316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:32.726704 17316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:32.727903 17316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:32.735983 17324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.736871 17322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.737524 17316 server_base.cc:1034] running on GCE node
W20240503 08:00:32.738188 17321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.738582 17316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:32.739056 17316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:32.740200 17316 hybrid_clock.cc:648] HybridClock initialized: now 1714723232740190 us; error 24 us; skew 500 ppm
I20240503 08:00:32.741967 17316 webserver.cc:460] Webserver started at http://127.0.99.130:39837/ using document root <none> and password file <none>
I20240503 08:00:32.742460 17316 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:32.742511 17316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:32.742672 17316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:32.744159 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.744557 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.747535 17316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:00:32.749480 17330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.750401 17316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:32.750485 17316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.750562 17316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:32.768747 17316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:32.769364 17316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:32.769498 17316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:32.769999 17316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:32.770833 17316 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:32.770874 17316 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.770915 17316 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:32.770941 17316 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.778910 17316 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43225
I20240503 08:00:32.778936 17443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43225 every 8 connection(s)
I20240503 08:00:32.779789 17316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:00:32.783265 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 17316
I20240503 08:00:32.783382 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:00:32.785673 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:32.788385 17444 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:32.788594 17444 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:32.789124 17444 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:32.820699 17123 ts_manager.cc:194] Registered new tserver with Master: f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:32.821440 17123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:58943
W20240503 08:00:32.887647 17447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:32.887851 17447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:32.887909 17447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:32.890990 17447 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:32.891057 17447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:32.891139 17447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 08:00:32.894644 17447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.17447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:32.895498 17447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:32.896553 17447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:32.904824 17456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.905695 17453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:32.906325 17454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:32.907161 17447 server_base.cc:1034] running on GCE node
I20240503 08:00:32.907522 17447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:32.908056 17447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:32.909221 17447 hybrid_clock.cc:648] HybridClock initialized: now 1714723232909200 us; error 38 us; skew 500 ppm
I20240503 08:00:32.911164 17447 webserver.cc:460] Webserver started at http://127.0.99.131:44223/ using document root <none> and password file <none>
I20240503 08:00:32.911724 17447 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:32.911782 17447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:32.911947 17447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:32.913571 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7af8f2a481294a2cb93e5761947978cc"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.914031 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7af8f2a481294a2cb93e5761947978cc"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.917152 17447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:00:32.919219 17462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.920163 17447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:32.920281 17447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7af8f2a481294a2cb93e5761947978cc"
format_stamp: "Formatted at 2024-05-03 08:00:32 on dist-test-slave-7b05"
I20240503 08:00:32.920358 17447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:32.936941 17447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:32.937551 17447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:32.937696 17447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:32.938191 17447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:32.939005 17447 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:32.939050 17447 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.939100 17447 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:32.939127 17447 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:32.946981 17447 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:39647
I20240503 08:00:32.947021 17575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:39647 every 8 connection(s)
I20240503 08:00:32.947824 17447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:00:32.951416 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 17447
I20240503 08:00:32.951522 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:00:32.953505 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:32.956188 17576 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:32.956370 17576 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:32.956837 17576 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:32.995246 17123 ts_manager.cc:194] Registered new tserver with Master: 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:32.996163 17123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:47159
W20240503 08:00:33.054179 17579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:33.054355 17579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:33.054394 17579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:33.057284 17579 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:33.057336 17579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:33.057397 17579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 08:00:33.060786 17579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.17579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:33.061527 17579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:33.062492 17579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:33.070456 17588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.070991 17586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.071877 17585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:33.072590 17579 server_base.cc:1034] running on GCE node
I20240503 08:00:33.072927 17579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:33.073457 17579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:33.074623 17579 hybrid_clock.cc:648] HybridClock initialized: now 1714723233074606 us; error 31 us; skew 500 ppm
I20240503 08:00:33.076318 17579 webserver.cc:460] Webserver started at http://127.0.99.132:38759/ using document root <none> and password file <none>
I20240503 08:00:33.076846 17579 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:33.076900 17579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:33.077090 17579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:33.078708 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0ce150b975ff4040bedde6319d5cf3d5"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.079152 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0ce150b975ff4040bedde6319d5cf3d5"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.082140 17579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:00:33.084132 17594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.085026 17579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:33.085141 17579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0ce150b975ff4040bedde6319d5cf3d5"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.085230 17579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:33.099658 17579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:33.100160 17579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:33.100301 17579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:33.100770 17579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:33.101713 17579 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:33.101770 17579 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.101823 17579 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:33.101851 17579 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.109753 17579 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:34863
I20240503 08:00:33.109778 17707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:34863 every 8 connection(s)
I20240503 08:00:33.110599 17579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:00:33.118137 17708 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:33.118321 17708 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:33.118753 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 17579
I20240503 08:00:33.118840 17708 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:33.118866 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:00:33.122609 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:33.150760 17123 ts_manager.cc:194] Registered new tserver with Master: 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863)
I20240503 08:00:33.151643 17123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:52691
W20240503 08:00:33.221812 17712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:33.221982 17712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:33.222020 17712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:33.224958 17712 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:33.225010 17712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:33.225097 17712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 08:00:33.228452 17712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.17712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:33.229167 17712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:33.230118 17712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:33.238163 17720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.238691 17717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.239254 17718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:33.240298 17712 server_base.cc:1034] running on GCE node
I20240503 08:00:33.240648 17712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:33.241183 17712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:33.242362 17712 hybrid_clock.cc:648] HybridClock initialized: now 1714723233242343 us; error 31 us; skew 500 ppm
I20240503 08:00:33.243942 17712 webserver.cc:460] Webserver started at http://127.0.99.133:35807/ using document root <none> and password file <none>
I20240503 08:00:33.244421 17712 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:33.244472 17712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:33.244613 17712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:33.246117 17712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "84305d92b082406380b107114e4cd61b"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.246526 17712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "84305d92b082406380b107114e4cd61b"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.249508 17712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:00:33.251345 17726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.252216 17712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:33.252307 17712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "84305d92b082406380b107114e4cd61b"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.252372 17712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:33.277333 17712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:33.277823 17712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:33.277943 17712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:33.278453 17712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:33.279266 17712 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:33.279311 17712 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.279345 17712 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:33.279366 17712 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.286566 17712 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:38825
I20240503 08:00:33.286599 17839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:38825 every 8 connection(s)
I20240503 08:00:33.287381 17712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:00:33.295174 17840 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:33.295364 17840 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:33.295799 17840 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:33.297528 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 17712
I20240503 08:00:33.297621 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:00:33.299926 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:33.327208 17123 ts_manager.cc:194] Registered new tserver with Master: 84305d92b082406380b107114e4cd61b (127.0.99.133:38825)
I20240503 08:00:33.327894 17123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:48937
W20240503 08:00:33.407538 17844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:33.407704 17844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:33.407742 17844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:33.410718 17844 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:33.410773 17844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:33.410840 17844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 08:00:33.414247 17844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.17844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:33.414986 17844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:33.415973 17844 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:33.423992 17852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.424549 17849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.425441 17850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:33.426203 17844 server_base.cc:1034] running on GCE node
I20240503 08:00:33.426527 17844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:33.426987 17844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:33.428125 17844 hybrid_clock.cc:648] HybridClock initialized: now 1714723233428115 us; error 27 us; skew 500 ppm
I20240503 08:00:33.429780 17844 webserver.cc:460] Webserver started at http://127.0.99.134:40189/ using document root <none> and password file <none>
I20240503 08:00:33.430253 17844 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:33.430310 17844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:33.430461 17844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:33.431928 17844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "78e52cf3c8fa435bb5ca36061227ac41"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.432353 17844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "78e52cf3c8fa435bb5ca36061227ac41"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.435386 17844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 08:00:33.437330 17858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.438241 17844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:00:33.438339 17844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "78e52cf3c8fa435bb5ca36061227ac41"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.438410 17844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:33.451362 17844 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:33.451839 17844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:33.451969 17844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:33.452422 17844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:33.453224 17844 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:33.453269 17844 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.453325 17844 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:33.453356 17844 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.461313 17844 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:35175
I20240503 08:00:33.461349 17971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:35175 every 8 connection(s)
I20240503 08:00:33.462091 17844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:00:33.468125 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 17844
I20240503 08:00:33.468226 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:00:33.469851 17972 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:33.470059 17972 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:33.470314 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:33.470608 17972 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:33.507037 17123 ts_manager.cc:194] Registered new tserver with Master: 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:33.507934 17123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:40919
W20240503 08:00:33.571329 17975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:33.571494 17975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:33.571539 17975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:33.574424 17975 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:33.574474 17975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:33.574540 17975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 08:00:33.577955 17975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.17975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:33.578668 17975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:33.579630 17975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:33.587664 17984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.588196 17982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.589069 17981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:33.589748 17975 server_base.cc:1034] running on GCE node
I20240503 08:00:33.590060 17975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:33.590533 17975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:33.591670 17975 hybrid_clock.cc:648] HybridClock initialized: now 1714723233591659 us; error 24 us; skew 500 ppm
I20240503 08:00:33.593248 17975 webserver.cc:460] Webserver started at http://127.0.99.135:33803/ using document root <none> and password file <none>
I20240503 08:00:33.593734 17975 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:33.593787 17975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:33.593936 17975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:33.595395 17975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.595810 17975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.598695 17975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20240503 08:00:33.600570 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.601461 17975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:33.601557 17975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.601627 17975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:33.617564 17975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:33.617992 17975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:33.618117 17975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:33.618542 17975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:33.619263 17975 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:33.619307 17975 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.619359 17975 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:33.619382 17975 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.626632 17975 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41359
I20240503 08:00:33.626670 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41359 every 8 connection(s)
I20240503 08:00:33.627393 17975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:00:33.634918 18104 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:33.635103 18104 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:33.635397 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 17975
I20240503 08:00:33.635507 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:00:33.635584 18104 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:33.637535 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:33.654980 17312 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:33.684046 17124 ts_manager.cc:194] Registered new tserver with Master: d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:33.684937 17124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:38505
W20240503 08:00:33.736754 18107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:33.736907 18107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:33.736943 18107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:33.739794 18107 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:33.739846 18107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:33.739905 18107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 08:00:33.743359 18107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.18107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:33.744019 18107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:33.744922 18107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:33.752516 18116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.753021 18113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.753631 18114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:33.754666 18107 server_base.cc:1034] running on GCE node
I20240503 08:00:33.754987 18107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:33.755458 18107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:33.756599 18107 hybrid_clock.cc:648] HybridClock initialized: now 1714723233756581 us; error 35 us; skew 500 ppm
I20240503 08:00:33.758242 18107 webserver.cc:460] Webserver started at http://127.0.99.136:37533/ using document root <none> and password file <none>
I20240503 08:00:33.758738 18107 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:33.758790 18107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:33.758939 18107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:33.760394 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "b65378a282b9443484843de0b8906df9"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.760829 18107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "b65378a282b9443484843de0b8906df9"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.763760 18107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 08:00:33.765625 18122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.766489 18107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:33.766579 18107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b65378a282b9443484843de0b8906df9"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.766651 18107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:33.786861 18107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:33.787400 18107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:33.787540 18107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:33.788072 18107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:33.788931 18107 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:33.788976 18107 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.789027 18107 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:33.789052 18107 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.796525 18107 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41085
I20240503 08:00:33.796545 18235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41085 every 8 connection(s)
I20240503 08:00:33.797400 18107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:00:33.802197 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 18107
I20240503 08:00:33.802310 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:00:33.804329 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.137:0
--local_ip_for_outbound_sockets=127.0.99.137
--webserver_interface=127.0.99.137
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:33.805617 18236 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:33.805804 18236 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:33.806257 18236 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:33.823479 17444 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:33.838871 17124 ts_manager.cc:194] Registered new tserver with Master: b65378a282b9443484843de0b8906df9 (127.0.99.136:41085)
I20240503 08:00:33.839521 17124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:58511
W20240503 08:00:33.906495 18239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:33.906689 18239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:33.906733 18239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:33.909732 18239 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:33.909788 18239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:33.909868 18239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.137
I20240503 08:00:33.913385 18239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.137:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.0.99.137
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.18239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.137
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:33.914168 18239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:33.915179 18239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:33.923312 18245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.924175 18246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:33.924785 18248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:33.925439 18239 server_base.cc:1034] running on GCE node
I20240503 08:00:33.925762 18239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:33.926250 18239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:33.927459 18239 hybrid_clock.cc:648] HybridClock initialized: now 1714723233927442 us; error 28 us; skew 500 ppm
I20240503 08:00:33.929188 18239 webserver.cc:460] Webserver started at http://127.0.99.137:34845/ using document root <none> and password file <none>
I20240503 08:00:33.929697 18239 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:33.929745 18239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:33.929883 18239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:33.931385 18239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "0665f4f85b984a8bba213ad479a88bad"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.931778 18239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "0665f4f85b984a8bba213ad479a88bad"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.934815 18239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 08:00:33.936784 18254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.937749 18239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 08:00:33.937837 18239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "0665f4f85b984a8bba213ad479a88bad"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:33.937914 18239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:33.950376 18239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:33.950884 18239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:33.951015 18239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:33.951475 18239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:33.952265 18239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:33.952307 18239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.952345 18239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:33.952373 18239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:33.960182 18239 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.137:44583
I20240503 08:00:33.960203 18367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.137:44583 every 8 connection(s)
I20240503 08:00:33.960999 18239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 08:00:33.968433 18368 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:33.968618 18368 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:33.969079 18368 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:33.969897 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 18239
I20240503 08:00:33.970012 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240503 08:00:33.973259 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.138:0
--local_ip_for_outbound_sockets=127.0.99.138
--webserver_interface=127.0.99.138
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:33.998358 17576 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:34.001596 17124 ts_manager.cc:194] Registered new tserver with Master: 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583)
I20240503 08:00:34.002378 17124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.137:40601
W20240503 08:00:34.074394 18372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:34.074576 18372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:34.074615 18372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:34.077637 18372 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:34.077692 18372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:34.077754 18372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.138
I20240503 08:00:34.081216 18372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.138:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.0.99.138
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.18372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.138
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:34.081974 18372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:34.082959 18372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:34.091253 18380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:34.092185 18378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:34.092723 18377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:34.093307 18372 server_base.cc:1034] running on GCE node
I20240503 08:00:34.093653 18372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:34.094158 18372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:34.095300 18372 hybrid_clock.cc:648] HybridClock initialized: now 1714723234095288 us; error 27 us; skew 500 ppm
I20240503 08:00:34.097009 18372 webserver.cc:460] Webserver started at http://127.0.99.138:43159/ using document root <none> and password file <none>
I20240503 08:00:34.097544 18372 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:34.097599 18372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:34.097757 18372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:00:34.099308 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data/instance:
uuid: "ab18888a9724446b9ee4fb9efe811b09"
format_stamp: "Formatted at 2024-05-03 08:00:34 on dist-test-slave-7b05"
I20240503 08:00:34.099757 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal/instance:
uuid: "ab18888a9724446b9ee4fb9efe811b09"
format_stamp: "Formatted at 2024-05-03 08:00:34 on dist-test-slave-7b05"
I20240503 08:00:34.102772 18372 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:00:34.104874 18386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:34.111609 18372 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20240503 08:00:34.111730 18372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "ab18888a9724446b9ee4fb9efe811b09"
format_stamp: "Formatted at 2024-05-03 08:00:34 on dist-test-slave-7b05"
I20240503 08:00:34.111822 18372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:34.123056 18372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:34.123570 18372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:34.123705 18372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:34.124164 18372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:34.124971 18372 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:34.125013 18372 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:34.125051 18372 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:34.125103 18372 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:34.132344 18372 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.138:33637
I20240503 08:00:34.132387 18499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.138:33637 every 8 connection(s)
I20240503 08:00:34.133200 18372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 08:00:34.138163 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 18372
I20240503 08:00:34.138276 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal/instance
I20240503 08:00:34.141347 18500 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:34.141521 18500 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:34.141953 18500 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:34.153946 17708 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:34.173801 17124 ts_manager.cc:194] Registered new tserver with Master: ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138:33637)
I20240503 08:00:34.174459 17124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.138:54993
I20240503 08:00:34.175731 398 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 08:00:34.184254 398 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240503 08:00:34.184327 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 18239
I20240503 08:00:34.189177 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 18372
I20240503 08:00:34.193706 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu with pid 18107
I20240503 08:00:34.330217 17840 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:34.510151 17972 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:34.686952 18104 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:37.949159 17123 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47184:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:37.952147 17123 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:00:37.969748 17906 tablet_service.cc:1467] Processing CreateTablet for tablet 08e9b83cd82f4d82ae552501be742896 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.969748 17774 tablet_service.cc:1467] Processing CreateTablet for tablet 08e9b83cd82f4d82ae552501be742896 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.970657 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e9b83cd82f4d82ae552501be742896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.970657 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e9b83cd82f4d82ae552501be742896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.972218 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 08e9b83cd82f4d82ae552501be742896 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.973320 17508 tablet_service.cc:1467] Processing CreateTablet for tablet 70f073cc41d54638a2c51746b94a5ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.973336 17772 tablet_service.cc:1467] Processing CreateTablet for tablet 70f073cc41d54638a2c51746b94a5ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.973577 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f073cc41d54638a2c51746b94a5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.973592 17904 tablet_service.cc:1467] Processing CreateTablet for tablet 70f073cc41d54638a2c51746b94a5ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.973793 17904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f073cc41d54638a2c51746b94a5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.973029 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e9b83cd82f4d82ae552501be742896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.974987 17246 tablet_service.cc:1467] Processing CreateTablet for tablet 08e9b83cd82f4d82ae552501be742896 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.975732 17246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e9b83cd82f4d82ae552501be742896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.973336 17773 tablet_service.cc:1467] Processing CreateTablet for tablet 42be6135d7654954b6886c32dfc688c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.977303 17773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be6135d7654954b6886c32dfc688c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.977298 17905 tablet_service.cc:1467] Processing CreateTablet for tablet 42be6135d7654954b6886c32dfc688c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.977540 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be6135d7654954b6886c32dfc688c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.977799 17244 tablet_service.cc:1467] Processing CreateTablet for tablet 70f073cc41d54638a2c51746b94a5ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.978029 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f073cc41d54638a2c51746b94a5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.977990 17245 tablet_service.cc:1467] Processing CreateTablet for tablet 42be6135d7654954b6886c32dfc688c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.978180 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be6135d7654954b6886c32dfc688c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.978329 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f073cc41d54638a2c51746b94a5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.979748 17509 tablet_service.cc:1467] Processing CreateTablet for tablet 42be6135d7654954b6886c32dfc688c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.979955 17509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be6135d7654954b6886c32dfc688c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.984028 18037 tablet_service.cc:1467] Processing CreateTablet for tablet 42be6135d7654954b6886c32dfc688c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.984440 18038 tablet_service.cc:1467] Processing CreateTablet for tablet 08e9b83cd82f4d82ae552501be742896 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.984053 18036 tablet_service.cc:1467] Processing CreateTablet for tablet 70f073cc41d54638a2c51746b94a5ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.984778 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be6135d7654954b6886c32dfc688c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.985211 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e9b83cd82f4d82ae552501be742896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.985824 18530 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:37.986310 18036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f073cc41d54638a2c51746b94a5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.986788 18529 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:37.987761 18530 tablet_bootstrap.cc:654] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:37.988597 18528 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:37.988814 18529 tablet_bootstrap.cc:654] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:37.990415 18529 log.cc:824] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:37.990743 18531 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:37.991012 18530 log.cc:824] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:37.991575 18527 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:37.991956 18528 tablet_bootstrap.cc:654] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:37.993288 18527 tablet_bootstrap.cc:654] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:37.993606 18531 tablet_bootstrap.cc:654] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:37.993960 18527 log.cc:824] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:37.994266 18531 log.cc:824] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:37.995533 18529 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:37.995669 18531 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:37.995692 18529 ts_tablet_manager.cc:1367] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:00:37.995808 18531 ts_tablet_manager.cc:1367] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:00:37.995746 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 08e9b83cd82f4d82ae552501be742896 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.996529 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e9b83cd82f4d82ae552501be742896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.997320 17377 tablet_service.cc:1467] Processing CreateTablet for tablet 42be6135d7654954b6886c32dfc688c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.997535 17377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be6135d7654954b6886c32dfc688c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.997311 17641 tablet_service.cc:1467] Processing CreateTablet for tablet 42be6135d7654954b6886c32dfc688c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:37.998005 17641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be6135d7654954b6886c32dfc688c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:37.998945 18531 raft_consensus.cc:348] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:37.999190 18531 raft_consensus.cc:374] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:37.999241 18531 raft_consensus.cc:729] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:37.999528 18529 raft_consensus.cc:348] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:37.999714 18529 raft_consensus.cc:374] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:37.999672 18531 consensus_queue.cc:260] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:37.999766 18529 raft_consensus.cc:729] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.000134 18529 consensus_queue.cc:260] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.000442 18531 ts_tablet_manager.cc:1398] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:00:38.000876 18529 ts_tablet_manager.cc:1398] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:00:38.001232 18529 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:38.001325 17376 tablet_service.cc:1467] Processing CreateTablet for tablet 70f073cc41d54638a2c51746b94a5ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.001332 17640 tablet_service.cc:1467] Processing CreateTablet for tablet 70f073cc41d54638a2c51746b94a5ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.001528 17376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f073cc41d54638a2c51746b94a5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.001542 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f073cc41d54638a2c51746b94a5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.002022 18529 tablet_bootstrap.cc:654] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:37.996977 17642 tablet_service.cc:1467] Processing CreateTablet for tablet 08e9b83cd82f4d82ae552501be742896 (DEFAULT_TABLE table=rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.003813 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e9b83cd82f4d82ae552501be742896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.006012 18531 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:38.006213 18528 log.cc:824] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:38.006835 18539 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:38.007822 18527 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:38.007884 18530 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:38.007974 18527 ts_tablet_manager.cc:1367] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 08:00:38.008008 18530 ts_tablet_manager.cc:1367] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240503 08:00:38.008033 18531 tablet_bootstrap.cc:654] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.008576 18539 tablet_bootstrap.cc:654] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.008968 18531 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:38.009032 18531 ts_tablet_manager.cc:1367] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:38.009286 18539 log.cc:824] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:38.009507 18531 raft_consensus.cc:348] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.009608 18531 raft_consensus.cc:374] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.009639 18531 raft_consensus.cc:729] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.009742 18531 consensus_queue.cc:260] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.009989 18531 ts_tablet_manager.cc:1398] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.010129 18531 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:38.010241 18540 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:38.010732 18529 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:38.010829 18529 ts_tablet_manager.cc:1367] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 08:00:38.010905 18531 tablet_bootstrap.cc:654] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.011330 18529 raft_consensus.cc:348] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.011451 18529 raft_consensus.cc:374] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.011492 18529 raft_consensus.cc:729] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.011358 18530 raft_consensus.cc:348] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.011600 18529 consensus_queue.cc:260] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.011672 18530 raft_consensus.cc:374] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.011811 18530 raft_consensus.cc:729] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.011853 18529 ts_tablet_manager.cc:1398] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.011811 18527 raft_consensus.cc:348] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.012004 18529 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:38.012073 18527 raft_consensus.cc:374] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.012148 18527 raft_consensus.cc:729] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.012311 18530 consensus_queue.cc:260] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.012538 18527 consensus_queue.cc:260] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.012805 18529 tablet_bootstrap.cc:654] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.013371 18527 ts_tablet_manager.cc:1398] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 08:00:38.013700 18527 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:38.013731 18530 ts_tablet_manager.cc:1398] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:00:38.014596 18527 tablet_bootstrap.cc:654] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.014998 18540 tablet_bootstrap.cc:654] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.015301 18530 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:38.015659 18540 log.cc:824] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:38.016201 18530 tablet_bootstrap.cc:654] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.017015 18531 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:38.017120 18531 ts_tablet_manager.cc:1367] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 08:00:38.017203 18539 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:38.017341 18539 ts_tablet_manager.cc:1367] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:00:38.017446 18530 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:38.017513 18530 ts_tablet_manager.cc:1367] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:38.017506 18531 raft_consensus.cc:348] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.017611 18531 raft_consensus.cc:374] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.017639 18531 raft_consensus.cc:729] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.017740 18531 consensus_queue.cc:260] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.017992 18531 ts_tablet_manager.cc:1398] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.018030 18529 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:38.017963 18530 raft_consensus.cc:348] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.018067 18530 raft_consensus.cc:374] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.018095 18529 ts_tablet_manager.cc:1367] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:00:38.018096 18530 raft_consensus.cc:729] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.018203 18530 consensus_queue.cc:260] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.018517 18530 ts_tablet_manager.cc:1398] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.018458 18529 raft_consensus.cc:348] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.018554 18529 raft_consensus.cc:374] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.018581 18529 raft_consensus.cc:729] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.018664 18530 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:38.018687 18529 consensus_queue.cc:260] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.019613 18529 ts_tablet_manager.cc:1398] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.020102 18530 tablet_bootstrap.cc:654] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.020116 18539 raft_consensus.cc:348] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.020304 18539 raft_consensus.cc:374] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.020370 18539 raft_consensus.cc:729] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.020743 18539 consensus_queue.cc:260] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.021483 18539 ts_tablet_manager.cc:1398] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:00:38.021766 18539 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:38.022289 18527 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:38.022362 18527 ts_tablet_manager.cc:1367] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:00:38.022575 18539 tablet_bootstrap.cc:654] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.022847 18528 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:38.022805 18527 raft_consensus.cc:348] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.022902 18527 raft_consensus.cc:374] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.022930 18527 raft_consensus.cc:729] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.023092 18528 ts_tablet_manager.cc:1367] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240503 08:00:38.023063 18527 consensus_queue.cc:260] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.023303 18527 ts_tablet_manager.cc:1398] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.023447 18527 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:38.023898 18540 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:38.024036 18540 ts_tablet_manager.cc:1367] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 08:00:38.024211 18527 tablet_bootstrap.cc:654] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.026430 18528 raft_consensus.cc:348] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.026628 18528 raft_consensus.cc:374] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.026690 18528 raft_consensus.cc:729] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.027072 18527 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:38.027153 18527 ts_tablet_manager.cc:1367] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:38.027103 18539 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:38.027078 18528 consensus_queue.cc:260] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.027154 18540 raft_consensus.cc:348] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.027331 18540 raft_consensus.cc:374] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.027289 18539 ts_tablet_manager.cc:1367] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:00:38.027387 18540 raft_consensus.cc:729] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.027521 18527 raft_consensus.cc:348] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.027634 18527 raft_consensus.cc:374] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.027671 18527 raft_consensus.cc:729] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.027765 18528 ts_tablet_manager.cc:1398] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:00:38.027784 18527 consensus_queue.cc:260] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.027791 18540 consensus_queue.cc:260] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.028059 18528 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:38.028052 18527 ts_tablet_manager.cc:1398] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.028479 18540 ts_tablet_manager.cc:1398] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 08:00:38.028744 18540 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:38.028893 18528 tablet_bootstrap.cc:654] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.029580 18540 tablet_bootstrap.cc:654] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.029980 18530 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:38.030056 18530 ts_tablet_manager.cc:1367] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 08:00:38.030445 18530 raft_consensus.cc:348] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.030524 18540 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:38.030558 18530 raft_consensus.cc:374] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.030593 18530 raft_consensus.cc:729] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.030593 18540 ts_tablet_manager.cc:1367] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:38.030704 18530 consensus_queue.cc:260] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.030901 18539 raft_consensus.cc:348] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.030977 18530 ts_tablet_manager.cc:1398] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.030997 18539 raft_consensus.cc:374] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.031025 18539 raft_consensus.cc:729] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.031105 18540 raft_consensus.cc:348] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.031205 18540 raft_consensus.cc:374] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.031159 18539 consensus_queue.cc:260] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.031237 18540 raft_consensus.cc:729] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.031355 18540 consensus_queue.cc:260] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.031402 18539 ts_tablet_manager.cc:1398] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:38.031661 18540 ts_tablet_manager.cc:1398] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.031786 18539 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:38.031818 18540 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:38.032608 18540 tablet_bootstrap.cc:654] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.033109 18539 tablet_bootstrap.cc:654] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
W20240503 08:00:38.033715 17445 tablet.cc:2369] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:00:38.033829 18540 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:38.033896 18540 ts_tablet_manager.cc:1367] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:38.034263 18540 raft_consensus.cc:348] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.034366 18540 raft_consensus.cc:374] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.034400 18540 raft_consensus.cc:729] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.034518 18540 consensus_queue.cc:260] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.034777 18540 ts_tablet_manager.cc:1398] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.035264 18528 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:38.035331 18528 ts_tablet_manager.cc:1367] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:00:38.035753 18528 raft_consensus.cc:348] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.035851 18528 raft_consensus.cc:374] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.035879 18528 raft_consensus.cc:729] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.036010 18539 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:38.035984 18528 consensus_queue.cc:260] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.036098 18539 ts_tablet_manager.cc:1367] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:38.036235 18528 ts_tablet_manager.cc:1398] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.036384 18528 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:38.036468 18539 raft_consensus.cc:348] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.036564 18539 raft_consensus.cc:374] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.036594 18539 raft_consensus.cc:729] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.036722 18539 consensus_queue.cc:260] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.037040 18539 ts_tablet_manager.cc:1398] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.037206 18528 tablet_bootstrap.cc:654] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.038237 18528 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:38.038319 18528 ts_tablet_manager.cc:1367] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:00:38.038343 18553 raft_consensus.cc:482] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.038466 18553 raft_consensus.cc:504] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.038682 18528 raft_consensus.cc:348] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.038794 18528 raft_consensus.cc:374] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.038834 18528 raft_consensus.cc:729] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.038935 18528 consensus_queue.cc:260] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.039187 18528 ts_tablet_manager.cc:1398] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:38.039783 18553 leader_election.cc:288] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
W20240503 08:00:38.040967 17841 tablet.cc:2369] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:00:38.041683 18536 raft_consensus.cc:482] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.041837 18536 raft_consensus.cc:504] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.043051 18536 leader_election.cc:288] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:38.043864 18557 raft_consensus.cc:482] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.043979 18557 raft_consensus.cc:504] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.045344 18557 leader_election.cc:288] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:38.046468 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:38.046713 17266 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:38.046857 18545 raft_consensus.cc:482] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.046980 18545 raft_consensus.cc:504] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.047554 17926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:38.047719 17926 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:38.048439 18545 leader_election.cc:288] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:38.048626 18537 raft_consensus.cc:482] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.048705 18537 raft_consensus.cc:504] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.052237 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:38.052448 17794 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:38.052938 17333 leader_election.cc:302] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:38.053241 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:38.053292 18553 raft_consensus.cc:2781] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:38.053303 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:38.053354 18553 raft_consensus.cc:482] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:38.053391 18553 raft_consensus.cc:3037] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.053386 17794 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 0.
I20240503 08:00:38.053386 18058 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:38.054391 18553 raft_consensus.cc:504] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.054917 18553 leader_election.cc:288] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:38.055397 17926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:38.055495 17926 raft_consensus.cc:3037] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.055493 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:38.055594 18058 raft_consensus.cc:3037] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.056453 17926 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:38.056492 18058 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:38.057049 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:38.057106 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:38.057178 17794 raft_consensus.cc:3037] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.057178 17266 raft_consensus.cc:3037] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.057791 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:38.057931 17662 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:38.058069 17794 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:38.058069 17266 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:38.058163 17661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:38.058254 17661 raft_consensus.cc:3037] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.058329 17333 leader_election.cc:302] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:38.058482 18553 raft_consensus.cc:2781] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:38.058640 18553 raft_consensus.cc:686] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Becoming Leader. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Running, Role: LEADER
I20240503 08:00:38.058862 17926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:38.058954 17926 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 0.
I20240503 08:00:38.058928 18553 consensus_queue.cc:237] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.059191 17661 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:38.059226 18537 leader_election.cc:288] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:38.062201 17122 catalog_manager.cc:5484] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:38.063162 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:38.063267 17266 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 0.
I20240503 08:00:38.063588 17991 leader_election.cc:302] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:38.063802 18536 raft_consensus.cc:2781] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:38.063848 18536 raft_consensus.cc:482] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:38.063882 18536 raft_consensus.cc:3037] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.064746 18536 raft_consensus.cc:504] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.065550 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:38.065801 17530 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:38.066000 18536 leader_election.cc:288] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [CANDIDATE]: Term 1 election: Requested vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:38.066084 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:38.066175 17529 raft_consensus.cc:3037] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.066426 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:38.066516 17794 raft_consensus.cc:3037] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.066572 17926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:38.066669 17926 raft_consensus.cc:3037] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.067116 17529 raft_consensus.cc:2454] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:38.067193 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:38.067283 17266 raft_consensus.cc:3037] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.067329 17794 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 1.
I20240503 08:00:38.067479 17926 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 1.
I20240503 08:00:38.067736 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:38.067873 17398 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 0.
I20240503 08:00:38.068111 17266 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 1.
I20240503 08:00:38.068138 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:38.068228 17397 raft_consensus.cc:3037] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.068835 17661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:38.068939 17661 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 0.
I20240503 08:00:38.069096 17397 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 1.
I20240503 08:00:38.069160 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:38.069244 17662 raft_consensus.cc:3037] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.069368 17992 leader_election.cc:302] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:38.069859 18536 raft_consensus.cc:2781] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:38.070029 18536 raft_consensus.cc:686] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 LEADER]: Becoming Leader. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Running, Role: LEADER
I20240503 08:00:38.070113 17662 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 1.
I20240503 08:00:38.070322 18536 consensus_queue.cc:237] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:38.070386 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:38.070466 17529 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 0.
I20240503 08:00:38.070613 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e9b83cd82f4d82ae552501be742896" candidate_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:38.070698 17530 raft_consensus.cc:3037] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.071518 17530 raft_consensus.cc:2454] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d502c5c7f12a4e39aa25cd57f92c403d in term 1.
I20240503 08:00:38.072906 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:38.073051 17266 raft_consensus.cc:2379] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ce150b975ff4040bedde6319d5cf3d5 in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:38.073391 17122 catalog_manager.cc:5484] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: term changed from 0 to 1, leader changed from <none> to d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135). New cstate: current_term: 1 leader_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:38.076607 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:38.078334 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:38.078395 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:38.078436 17662 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:38.078487 17794 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:38.079439 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:38.079561 18058 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:38.081501 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:38.081658 17530 raft_consensus.cc:2379] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ce150b975ff4040bedde6319d5cf3d5 in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:38.082852 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:38.082950 17794 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:38.083261 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:38.083350 17662 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:38.083568 17859 leader_election.cc:302] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:38.083796 18545 raft_consensus.cc:2781] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:38.083874 18545 raft_consensus.cc:482] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:38.083920 18545 raft_consensus.cc:3037] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.083961 17926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:38.084120 17926 raft_consensus.cc:2379] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ce150b975ff4040bedde6319d5cf3d5 in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:38.084487 17596 leader_election.cc:302] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5; no voters: 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298
I20240503 08:00:38.084689 18557 raft_consensus.cc:2726] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:00:38.084805 18545 raft_consensus.cc:504] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.085386 18545 leader_election.cc:288] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 election: Requested vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:38.085469 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:38.085563 17794 raft_consensus.cc:3037] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.085557 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:38.085649 18058 raft_consensus.cc:3037] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.086380 17794 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:38.086555 18058 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:38.086982 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:38.087074 17662 raft_consensus.cc:3037] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.087780 17662 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:38.088197 17859 leader_election.cc:302] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:38.088389 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:38.088527 18058 raft_consensus.cc:2379] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ce150b975ff4040bedde6319d5cf3d5 in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:38.088935 17926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:38.089027 18556 raft_consensus.cc:482] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.089141 18556 raft_consensus.cc:504] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.088932 18545 raft_consensus.cc:2781] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:38.089438 18545 raft_consensus.cc:686] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Becoming Leader. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Running, Role: LEADER
I20240503 08:00:38.089722 18545 consensus_queue.cc:237] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.090513 18556 leader_election.cc:288] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:38.092994 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:38.093132 18058 raft_consensus.cc:2379] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8eb6822290b046f88c861159aa9421ec in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:38.093259 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:38.093199 17122 catalog_manager.cc:5484] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:38.093350 17397 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:38.093688 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:38.093807 17530 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:38.093705 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:38.093940 17529 raft_consensus.cc:3037] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.094714 17529 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:38.095248 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:38.095347 17397 raft_consensus.cc:3037] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.096185 17397 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:38.096900 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be6135d7654954b6886c32dfc688c5" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:38.097043 17794 raft_consensus.cc:2379] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ce150b975ff4040bedde6319d5cf3d5 in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:38.097795 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:38.097924 17397 raft_consensus.cc:2379] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8eb6822290b046f88c861159aa9421ec in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:38.099135 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:38.099252 17794 raft_consensus.cc:2379] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7af8f2a481294a2cb93e5761947978cc in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:38.101265 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:38.101289 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:38.101387 17266 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:38.101413 17662 raft_consensus.cc:2379] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7af8f2a481294a2cb93e5761947978cc in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:38.101488 17926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:38.102505 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:38.102634 18058 raft_consensus.cc:2379] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7af8f2a481294a2cb93e5761947978cc in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:38.102653 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:38.102770 17529 raft_consensus.cc:2379] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8eb6822290b046f88c861159aa9421ec in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:38.103029 17199 leader_election.cc:302] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec; no voters: 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298
I20240503 08:00:38.103241 18537 raft_consensus.cc:3037] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.103471 17464 leader_election.cc:302] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 7af8f2a481294a2cb93e5761947978cc, 8eb6822290b046f88c861159aa9421ec; no voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d
I20240503 08:00:38.103690 18556 raft_consensus.cc:2726] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:00:38.104071 18537 raft_consensus.cc:2726] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:00:38.104815 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:38.104861 17265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:38.104939 17266 raft_consensus.cc:2454] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:38.105537 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f073cc41d54638a2c51746b94a5ece" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:38.105649 17397 raft_consensus.cc:2379] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7af8f2a481294a2cb93e5761947978cc in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
W20240503 08:00:38.112777 17313 tablet.cc:2369] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:38.114228 17709 tablet.cc:2369] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:38.130560 18105 tablet.cc:2369] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:38.201790 17577 tablet.cc:2369] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:38.215610 17973 tablet.cc:2369] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:00:38.216382 18625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:38.216579 18625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:38.219745 18625 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 08:00:38.306344 17662 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.307005 17794 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.307451 17793 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.307570 18613 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.308236 18613 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.308308 18536 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.308869 17529 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.309893 17926 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.310376 18615 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.311046 18615 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.311205 17661 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.311635 17397 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.312134 17265 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.312613 18615 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.312865 18615 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.314015 17925 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.314445 18058 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.315521 17530 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.315961 17265 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.316387 18553 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.316671 18553 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.316923 18553 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.317178 18553 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.319087 18058 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.319367 17398 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.319737 17530 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.319881 17792 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.320372 18545 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.320696 18545 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.320770 17660 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.320955 18545 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.321669 18536 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.325649 17265 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.329823 18617 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.330112 18617 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.330608 18617 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.340842 18654 mvcc.cc:205] Tried to move back new op lower bound from 7023506384146132992 to 7023506383219171328. Current Snapshot: MvccSnapshot[applied={T|T < 7023506384146132992}]
I20240503 08:00:38.342885 18687 mvcc.cc:205] Tried to move back new op lower bound from 7023506384100069376 to 7023506383139946496. Current Snapshot: MvccSnapshot[applied={T|T < 7023506384100069376}]
I20240503 08:00:38.351358 18653 mvcc.cc:205] Tried to move back new op lower bound from 7023506384146132992 to 7023506383219171328. Current Snapshot: MvccSnapshot[applied={T|T < 7023506384146132992}]
I20240503 08:00:38.356750 18691 mvcc.cc:205] Tried to move back new op lower bound from 7023506384146132992 to 7023506383219171328. Current Snapshot: MvccSnapshot[applied={T|T < 7023506384146132992}]
I20240503 08:00:38.382735 18681 mvcc.cc:205] Tried to move back new op lower bound from 7023506384100069376 to 7023506383139946496. Current Snapshot: MvccSnapshot[applied={T|T < 7023506384100069376}]
I20240503 08:00:38.391844 18673 mvcc.cc:205] Tried to move back new op lower bound from 7023506384100069376 to 7023506383139946496. Current Snapshot: MvccSnapshot[applied={T|T < 7023506384100069376}]
I20240503 08:00:38.409296 18679 mvcc.cc:205] Tried to move back new op lower bound from 7023506384195153920 to 7023506383139946496. Current Snapshot: MvccSnapshot[applied={T|T < 7023506384100069376}]
INSERT report
rows total: 2000
time total: 213.607 ms
time per row: 0.106803 ms
I20240503 08:00:38.476425 17116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47184:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:38.476905 17116 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:00:38.482164 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 8a03fdbe7f824674a351fd4598dc2c63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.482338 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 8a03fdbe7f824674a351fd4598dc2c63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.482470 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.482605 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.484488 17244 tablet_service.cc:1467] Processing CreateTablet for tablet 8a03fdbe7f824674a351fd4598dc2c63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.484597 18539 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:38.484742 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.484952 17906 tablet_service.cc:1467] Processing CreateTablet for tablet 8a03fdbe7f824674a351fd4598dc2c63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.485217 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.485666 18539 tablet_bootstrap.cc:654] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.486690 18528 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:38.487118 17377 tablet_service.cc:1467] Processing CreateTablet for tablet 5154e7f64d1542979f9363a429fa469c (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.487335 17377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154e7f64d1542979f9363a429fa469c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.487347 17641 tablet_service.cc:1467] Processing CreateTablet for tablet 8a03fdbe7f824674a351fd4598dc2c63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.487496 18528 tablet_bootstrap.cc:654] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.487613 17641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.487840 18539 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:38.487911 18539 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:00:38.488081 18529 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:38.488310 18539 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.488425 18539 raft_consensus.cc:374] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.488456 18539 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.488569 18539 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.488829 18539 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:38.488873 18529 tablet_bootstrap.cc:654] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.489269 18539 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:38.489343 17378 tablet_service.cc:1467] Processing CreateTablet for tablet ba0615c2fbc840c7bbdfda22caf45de2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.489373 17640 tablet_service.cc:1467] Processing CreateTablet for tablet 5154e7f64d1542979f9363a429fa469c (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.489526 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0615c2fbc840c7bbdfda22caf45de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.489588 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154e7f64d1542979f9363a429fa469c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.489799 17642 tablet_service.cc:1467] Processing CreateTablet for tablet ba0615c2fbc840c7bbdfda22caf45de2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.489985 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0615c2fbc840c7bbdfda22caf45de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.490002 18539 tablet_bootstrap.cc:654] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.491272 18540 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:38.491654 18527 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:38.491925 17244 tablet_service.cc:1467] Processing CreateTablet for tablet ba0615c2fbc840c7bbdfda22caf45de2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.492138 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0615c2fbc840c7bbdfda22caf45de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.492372 18540 tablet_bootstrap.cc:654] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.492311 18037 tablet_service.cc:1467] Processing CreateTablet for tablet 5154e7f64d1542979f9363a429fa469c (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.492419 18527 tablet_bootstrap.cc:654] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.492560 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154e7f64d1542979f9363a429fa469c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.493158 17508 tablet_service.cc:1467] Processing CreateTablet for tablet 5154e7f64d1542979f9363a429fa469c (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.493367 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154e7f64d1542979f9363a429fa469c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.494069 18038 tablet_service.cc:1467] Processing CreateTablet for tablet 8a03fdbe7f824674a351fd4598dc2c63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.494290 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.494468 18531 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:38.495199 18531 tablet_bootstrap.cc:654] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.495827 17246 tablet_service.cc:1467] Processing CreateTablet for tablet 5154e7f64d1542979f9363a429fa469c (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.496031 17246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154e7f64d1542979f9363a429fa469c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.496095 18036 tablet_service.cc:1467] Processing CreateTablet for tablet ba0615c2fbc840c7bbdfda22caf45de2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.496106 18531 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:38.496268 18531 ts_tablet_manager.cc:1367] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:38.496315 18036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0615c2fbc840c7bbdfda22caf45de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.496661 18531 raft_consensus.cc:348] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.496783 18531 raft_consensus.cc:374] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.496824 18531 raft_consensus.cc:729] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.496953 18531 consensus_queue.cc:260] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.497246 18531 ts_tablet_manager.cc:1398] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.497402 18531 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:38.497525 17772 tablet_service.cc:1467] Processing CreateTablet for tablet 8a03fdbe7f824674a351fd4598dc2c63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.497771 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.497746 17774 tablet_service.cc:1467] Processing CreateTablet for tablet ba0615c2fbc840c7bbdfda22caf45de2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.497960 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0615c2fbc840c7bbdfda22caf45de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.498140 18531 tablet_bootstrap.cc:654] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.498574 18528 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:38.498652 18528 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 08:00:38.499073 18528 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.499197 18528 raft_consensus.cc:374] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.499230 18528 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.499346 18528 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.499620 18528 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.499768 18528 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:38.499965 17773 tablet_service.cc:1467] Processing CreateTablet for tablet 5154e7f64d1542979f9363a429fa469c (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.500042 18530 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:38.500165 17773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154e7f64d1542979f9363a429fa469c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.500506 18528 tablet_bootstrap.cc:654] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.500731 18530 tablet_bootstrap.cc:654] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.501608 18529 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:38.501689 18529 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 08:00:38.501789 18540 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:38.501873 18540 ts_tablet_manager.cc:1367] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:00:38.502111 18529 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.502215 18529 raft_consensus.cc:374] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.502245 18529 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.502261 18540 raft_consensus.cc:348] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.502355 18530 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:38.502360 18540 raft_consensus.cc:374] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.502393 18540 raft_consensus.cc:729] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.502354 18529 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.502426 18530 ts_tablet_manager.cc:1367] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:38.502501 18540 consensus_queue.cc:260] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.502619 18529 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.502760 18529 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:38.502755 18540 ts_tablet_manager.cc:1398] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:38.502908 18540 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:38.502931 18530 raft_consensus.cc:348] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.503048 18530 raft_consensus.cc:374] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.503080 18530 raft_consensus.cc:729] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.503187 18530 consensus_queue.cc:260] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.503425 18530 ts_tablet_manager.cc:1398] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.503559 18529 tablet_bootstrap.cc:654] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.503564 18530 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:38.503670 18540 tablet_bootstrap.cc:654] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.504316 18530 tablet_bootstrap.cc:654] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.505474 18530 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:38.505787 17906 tablet_service.cc:1467] Processing CreateTablet for tablet ba0615c2fbc840c7bbdfda22caf45de2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.505858 17904 tablet_service.cc:1467] Processing CreateTablet for tablet 5154e7f64d1542979f9363a429fa469c (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.506011 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0615c2fbc840c7bbdfda22caf45de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.506286 17904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154e7f64d1542979f9363a429fa469c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.507880 18529 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:38.507956 18529 ts_tablet_manager.cc:1367] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:00:38.508008 18530 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:38.508329 18529 raft_consensus.cc:348] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.508380 18530 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.508440 18529 raft_consensus.cc:374] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.508463 18530 raft_consensus.cc:374] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.508468 18529 raft_consensus.cc:729] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.508492 18530 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.508606 18530 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.508599 18529 consensus_queue.cc:260] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.508869 18530 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.509006 18530 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:38.509539 18527 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:38.509618 18527 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20240503 08:00:38.509781 18530 tablet_bootstrap.cc:654] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.510002 18527 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.510092 18527 raft_consensus.cc:374] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.510128 18527 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.510236 18527 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.510469 18527 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.510623 18527 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:38.510812 18530 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:38.510876 18530 ts_tablet_manager.cc:1367] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:38.511242 18530 raft_consensus.cc:348] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.511337 18530 raft_consensus.cc:374] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.511366 18530 raft_consensus.cc:729] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.511420 18529 ts_tablet_manager.cc:1398] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:38.511443 18527 tablet_bootstrap.cc:654] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.511468 18530 consensus_queue.cc:260] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.511715 18530 ts_tablet_manager.cc:1398] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.512070 18540 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:38.512143 18540 ts_tablet_manager.cc:1367] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 08:00:38.513200 17510 tablet_service.cc:1467] Processing CreateTablet for tablet ba0615c2fbc840c7bbdfda22caf45de2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:38.513346 18529 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:38.513430 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0615c2fbc840c7bbdfda22caf45de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:38.514170 18529 tablet_bootstrap.cc:654] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.512550 18540 raft_consensus.cc:348] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.514654 18540 raft_consensus.cc:374] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.514698 18540 raft_consensus.cc:729] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.514829 18540 consensus_queue.cc:260] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.515053 18529 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:38.515134 18529 ts_tablet_manager.cc:1367] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:38.515296 18540 ts_tablet_manager.cc:1398] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:38.515509 18528 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:38.515591 18528 ts_tablet_manager.cc:1367] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 08:00:38.515597 18540 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:38.515942 18528 raft_consensus.cc:348] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.516028 18528 raft_consensus.cc:374] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.516055 18528 raft_consensus.cc:729] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.516161 18528 consensus_queue.cc:260] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.516381 18528 ts_tablet_manager.cc:1398] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:38.516520 18528 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:38.516515 18529 raft_consensus.cc:348] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.516628 18529 raft_consensus.cc:374] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.516659 18529 raft_consensus.cc:729] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.517311 18540 tablet_bootstrap.cc:654] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.517366 18528 tablet_bootstrap.cc:654] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.517379 18529 consensus_queue.cc:260] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.518657 18531 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:38.518743 18531 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 08:00:38.518839 18529 ts_tablet_manager.cc:1398] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:00:38.518936 18539 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:38.519021 18539 ts_tablet_manager.cc:1367] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20240503 08:00:38.519160 18531 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.519294 18531 raft_consensus.cc:374] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.519342 18531 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.519444 18539 raft_consensus.cc:348] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.519462 18531 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.519538 18539 raft_consensus.cc:374] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.519567 18539 raft_consensus.cc:729] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.519676 18539 consensus_queue.cc:260] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.519721 18531 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:00:38.519881 18531 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:38.519920 18539 ts_tablet_manager.cc:1398] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:38.520079 18539 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:38.520714 18531 tablet_bootstrap.cc:654] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.520831 18528 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:38.520892 18539 tablet_bootstrap.cc:654] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.520898 18528 ts_tablet_manager.cc:1367] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:38.521344 18528 raft_consensus.cc:348] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.521454 18528 raft_consensus.cc:374] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.521483 18528 raft_consensus.cc:729] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.521585 18528 consensus_queue.cc:260] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.521842 18528 ts_tablet_manager.cc:1398] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.523018 18527 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:38.523120 18527 ts_tablet_manager.cc:1367] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 08:00:38.523285 18531 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:38.523355 18531 ts_tablet_manager.cc:1367] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:38.523463 18539 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:38.523531 18539 ts_tablet_manager.cc:1367] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:38.523504 18527 raft_consensus.cc:348] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.523622 18527 raft_consensus.cc:374] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.523651 18527 raft_consensus.cc:729] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.523721 18531 raft_consensus.cc:348] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.523757 18527 consensus_queue.cc:260] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.523826 18531 raft_consensus.cc:374] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.523855 18531 raft_consensus.cc:729] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.523919 18539 raft_consensus.cc:348] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.523959 18531 consensus_queue.cc:260] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.524016 18539 raft_consensus.cc:374] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.524019 18527 ts_tablet_manager.cc:1398] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:38.524046 18539 raft_consensus.cc:729] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.524170 18527 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:38.524163 18539 consensus_queue.cc:260] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.524217 18531 ts_tablet_manager.cc:1398] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.524405 18539 ts_tablet_manager.cc:1398] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:38.524655 18540 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:38.524724 18540 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:00:38.524973 18527 tablet_bootstrap.cc:654] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:38.525296 18540 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.525983 18540 raft_consensus.cc:374] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.526026 18540 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.526165 18540 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.526535 18540 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:38.526561 18527 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:38.526624 18527 ts_tablet_manager.cc:1367] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:38.527186 18527 raft_consensus.cc:348] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.527310 18527 raft_consensus.cc:374] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:38.527351 18527 raft_consensus.cc:729] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:38.527463 18527 consensus_queue.cc:260] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.527693 18527 ts_tablet_manager.cc:1398] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:38.547394 18537 raft_consensus.cc:482] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.547511 18537 raft_consensus.cc:504] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.548041 18537 leader_election.cc:288] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:38.548260 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:38.548389 17397 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:38.548370 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:38.548492 18058 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:38.548497 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:38.548655 17793 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:38.548728 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:38.548728 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:38.548859 17530 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:38.548882 17662 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:38.549000 17199 leader_election.cc:302] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:38.549135 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:38.549201 18537 raft_consensus.cc:2781] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:38.549252 18537 raft_consensus.cc:482] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:38.549264 17925 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:38.549299 18537 raft_consensus.cc:3037] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.550063 18537 raft_consensus.cc:504] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.550504 18537 leader_election.cc:288] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 election: Requested vote from peers f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:38.550669 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:38.550771 17397 raft_consensus.cc:3037] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.550763 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:38.550845 17530 raft_consensus.cc:3037] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.550976 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:38.551052 17793 raft_consensus.cc:3037] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.551671 17397 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:38.551671 17530 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:38.551765 17793 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:38.551891 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:38.551891 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:38.551995 17925 raft_consensus.cc:3037] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.551995 18058 raft_consensus.cc:3037] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.552156 17199 leader_election.cc:302] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:38.552858 17925 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:38.552872 18058 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:38.553050 18537 raft_consensus.cc:2781] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:38.553112 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a03fdbe7f824674a351fd4598dc2c63" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:38.553202 17662 raft_consensus.cc:3037] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.553244 18537 raft_consensus.cc:686] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Becoming Leader. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Running, Role: LEADER
I20240503 08:00:38.553548 18537 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:38.554072 17662 raft_consensus.cc:2454] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:38.556310 17119 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eb6822290b046f88c861159aa9421ec (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:38.613250 18557 raft_consensus.cc:482] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.613376 18557 raft_consensus.cc:504] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.613978 18557 leader_election.cc:288] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:38.614329 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:38.614457 17793 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:38.614460 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:38.614452 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:38.614564 17266 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:38.614614 17530 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:38.614610 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:38.614729 17397 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:38.615222 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:38.615301 17595 leader_election.cc:302] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:38.615366 18058 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:38.615514 18557 raft_consensus.cc:2781] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:38.615567 18557 raft_consensus.cc:482] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:38.615608 18557 raft_consensus.cc:3037] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.615783 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:38.615936 17925 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:38.616611 18557 raft_consensus.cc:504] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.617218 18557 leader_election.cc:288] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 election: Requested vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:38.617292 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:38.617424 17793 raft_consensus.cc:3037] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.617414 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:38.617509 18058 raft_consensus.cc:3037] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.617578 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:38.617661 17266 raft_consensus.cc:3037] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.617731 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:38.617830 17530 raft_consensus.cc:3037] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.618302 17793 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:38.618463 18058 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:38.618499 17266 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:38.618670 17530 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:38.618932 17596 leader_election.cc:302] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:38.618963 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:38.619060 17397 raft_consensus.cc:3037] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.619165 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0615c2fbc840c7bbdfda22caf45de2" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:38.619277 17925 raft_consensus.cc:3037] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.619882 17397 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:38.620065 17925 raft_consensus.cc:2454] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:38.620074 18557 raft_consensus.cc:2781] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:38.620241 18557 raft_consensus.cc:686] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Running, Role: LEADER
I20240503 08:00:38.620533 18557 consensus_queue.cc:237] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.623055 17119 catalog_manager.cc:5484] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:38.669234 18556 raft_consensus.cc:482] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:38.669353 18556 raft_consensus.cc:504] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.669991 18556 leader_election.cc:288] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:38.670243 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:38.670338 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:38.670377 17662 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:38.670397 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:38.670397 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:38.670450 17397 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:38.670532 17793 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:38.670532 17925 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:38.670754 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:38.670881 17266 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:38.670866 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:38.670989 18058 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:38.670984 17465 leader_election.cc:302] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:38.671172 18556 raft_consensus.cc:2781] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:38.671233 18556 raft_consensus.cc:482] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:38.671272 18556 raft_consensus.cc:3037] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.672022 18556 raft_consensus.cc:504] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.672518 18556 leader_election.cc:288] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 election: Requested vote from peers 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:38.672700 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:38.672773 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:38.672773 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:38.672801 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:38.672839 17793 raft_consensus.cc:3037] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.672868 18058 raft_consensus.cc:3037] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.672888 17662 raft_consensus.cc:3037] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.672909 17925 raft_consensus.cc:3037] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.673713 17793 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:38.673713 17925 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:38.673770 18058 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:38.673770 17662 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:38.674016 17464 leader_election.cc:302] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:38.674170 18556 raft_consensus.cc:2781] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:38.674294 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:38.674328 18556 raft_consensus.cc:686] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Becoming Leader. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Running, Role: LEADER
I20240503 08:00:38.674324 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154e7f64d1542979f9363a429fa469c" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:38.674405 17397 raft_consensus.cc:3037] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.674418 17266 raft_consensus.cc:3037] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:38.674650 18556 consensus_queue.cc:237] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:38.675189 17397 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:38.675302 17266 raft_consensus.cc:2454] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:38.677049 17119 catalog_manager.cc:5484] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7af8f2a481294a2cb93e5761947978cc (127.0.99.131). New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } }
W20240503 08:00:38.790138 18751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:38.790354 18751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:38.793282 18751 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:00:38.865278 17266 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.865681 18058 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.866206 18556 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.866389 18749 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.867247 17925 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.867242 17793 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.867588 17398 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.868032 17266 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.868047 18556 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.868172 18557 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.868341 18556 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.869115 17397 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.869354 17926 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.869398 17662 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.869408 17530 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.869834 18058 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.870419 17792 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.870988 17793 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.871929 18749 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.872556 17662 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.873044 18747 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.875918 17530 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.876207 17926 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.876348 17396 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.876375 18782 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.876605 18745 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.876971 18745 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.877254 18745 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.877481 18745 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.877696 18745 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.878017 18747 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.878294 18747 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.878551 18747 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.878794 18747 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:38.882145 18058 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:38.884866 18537 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 180.201 ms
time per row: 0.0901005 ms
I20240503 08:00:39.016269 17122 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47184:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:39.016806 17122 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:00:39.022725 17246 tablet_service.cc:1467] Processing CreateTablet for tablet 143109561ad549d69a59992c02fe23af (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.023053 17246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143109561ad549d69a59992c02fe23af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.025193 17906 tablet_service.cc:1467] Processing CreateTablet for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.025451 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.025691 17904 tablet_service.cc:1467] Processing CreateTablet for tablet 143109561ad549d69a59992c02fe23af (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.025918 17904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143109561ad549d69a59992c02fe23af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.026540 17905 tablet_service.cc:1467] Processing CreateTablet for tablet 00708b0b8f2f4366944b244a9ba8c29f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.026746 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00708b0b8f2f4366944b244a9ba8c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.028028 18036 tablet_service.cc:1467] Processing CreateTablet for tablet 143109561ad549d69a59992c02fe23af (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.028276 18036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143109561ad549d69a59992c02fe23af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.028752 18817 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:39.029719 18817 tablet_bootstrap.cc:654] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.030225 18818 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:39.030328 18038 tablet_service.cc:1467] Processing CreateTablet for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.030530 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.030730 18817 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:39.030831 18817 ts_tablet_manager.cc:1367] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:39.031383 18818 tablet_bootstrap.cc:654] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.031337 18817 raft_consensus.cc:348] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.031471 18817 raft_consensus.cc:374] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.031508 18817 raft_consensus.cc:729] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.031642 18817 consensus_queue.cc:260] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.032021 18817 ts_tablet_manager.cc:1398] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.032217 18037 tablet_service.cc:1467] Processing CreateTablet for tablet 00708b0b8f2f4366944b244a9ba8c29f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.032414 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00708b0b8f2f4366944b244a9ba8c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.032733 18818 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:39.032816 18818 ts_tablet_manager.cc:1367] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:39.032949 18527 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:39.033439 18818 raft_consensus.cc:348] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.033564 18818 raft_consensus.cc:374] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.033599 18818 raft_consensus.cc:729] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.033726 18818 consensus_queue.cc:260] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.033799 18527 tablet_bootstrap.cc:654] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.034024 18818 ts_tablet_manager.cc:1398] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.034190 18818 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:39.035215 18818 tablet_bootstrap.cc:654] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.035708 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 143109561ad549d69a59992c02fe23af (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.035951 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143109561ad549d69a59992c02fe23af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.036141 17508 tablet_service.cc:1467] Processing CreateTablet for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.036407 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.036655 18527 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:39.036733 18527 ts_tablet_manager.cc:1367] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:39.037200 17641 tablet_service.cc:1467] Processing CreateTablet for tablet 143109561ad549d69a59992c02fe23af (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.037302 18527 raft_consensus.cc:348] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.037420 18527 raft_consensus.cc:374] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.037449 17641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143109561ad549d69a59992c02fe23af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.037453 18527 raft_consensus.cc:729] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.037585 18527 consensus_queue.cc:260] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.037876 18527 ts_tablet_manager.cc:1398] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.038980 17509 tablet_service.cc:1467] Processing CreateTablet for tablet 00708b0b8f2f4366944b244a9ba8c29f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.039216 17509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00708b0b8f2f4366944b244a9ba8c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.039268 18823 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:39.039661 17376 tablet_service.cc:1467] Processing CreateTablet for tablet 00708b0b8f2f4366944b244a9ba8c29f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.039909 17376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00708b0b8f2f4366944b244a9ba8c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.040222 18823 tablet_bootstrap.cc:654] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.041005 18527 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:39.041333 17642 tablet_service.cc:1467] Processing CreateTablet for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.041589 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.042038 17377 tablet_service.cc:1467] Processing CreateTablet for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.042104 18527 tablet_bootstrap.cc:654] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.042124 18826 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:39.042279 17377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.043123 18826 tablet_bootstrap.cc:654] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.043326 18818 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:39.043421 18818 ts_tablet_manager.cc:1367] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:00:39.043656 17640 tablet_service.cc:1467] Processing CreateTablet for tablet 00708b0b8f2f4366944b244a9ba8c29f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.043885 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00708b0b8f2f4366944b244a9ba8c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.043879 18818 raft_consensus.cc:348] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.043998 18818 raft_consensus.cc:374] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.044027 18818 raft_consensus.cc:729] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.044119 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 143109561ad549d69a59992c02fe23af (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.044152 18818 consensus_queue.cc:260] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.044337 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143109561ad549d69a59992c02fe23af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.044440 18818 ts_tablet_manager.cc:1398] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.044598 18818 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:39.045437 18818 tablet_bootstrap.cc:654] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.046321 17244 tablet_service.cc:1467] Processing CreateTablet for tablet 00708b0b8f2f4366944b244a9ba8c29f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.046602 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00708b0b8f2f4366944b244a9ba8c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.046669 18527 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:39.046736 18527 ts_tablet_manager.cc:1367] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:00:39.046762 18823 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:39.046840 18823 ts_tablet_manager.cc:1367] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:00:39.047142 18527 raft_consensus.cc:348] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.047255 18527 raft_consensus.cc:374] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.047286 18527 raft_consensus.cc:729] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.047271 18823 raft_consensus.cc:348] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.047385 18823 raft_consensus.cc:374] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.047420 18823 raft_consensus.cc:729] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.047396 18527 consensus_queue.cc:260] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.047541 18823 consensus_queue.cc:260] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.047767 18527 ts_tablet_manager.cc:1398] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.047807 18823 ts_tablet_manager.cc:1398] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.047968 18823 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:39.048389 18527 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:39.048756 18823 tablet_bootstrap.cc:654] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.046321 17246 tablet_service.cc:1467] Processing CreateTablet for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.049113 17246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.049229 18527 tablet_bootstrap.cc:654] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.050155 18527 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:39.050172 18823 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:39.050240 18527 ts_tablet_manager.cc:1367] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.050246 18823 ts_tablet_manager.cc:1367] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.050592 18823 raft_consensus.cc:348] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.050679 18823 raft_consensus.cc:374] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.050707 18823 raft_consensus.cc:729] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.050634 18527 raft_consensus.cc:348] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.050762 18527 raft_consensus.cc:374] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.050801 18527 raft_consensus.cc:729] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.050813 18823 consensus_queue.cc:260] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.050901 18527 consensus_queue.cc:260] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.051077 18823 ts_tablet_manager.cc:1398] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.051224 18823 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:39.051287 18527 ts_tablet_manager.cc:1398] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.052023 18823 tablet_bootstrap.cc:654] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.052227 17773 tablet_service.cc:1467] Processing CreateTablet for tablet 143109561ad549d69a59992c02fe23af (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.052479 17773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143109561ad549d69a59992c02fe23af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.052970 17774 tablet_service.cc:1467] Processing CreateTablet for tablet 00708b0b8f2f4366944b244a9ba8c29f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.053232 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00708b0b8f2f4366944b244a9ba8c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.053458 18823 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:39.053526 18823 ts_tablet_manager.cc:1367] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:39.053957 18823 raft_consensus.cc:348] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.054069 18823 raft_consensus.cc:374] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.054100 18823 raft_consensus.cc:729] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.054215 18823 consensus_queue.cc:260] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.054473 18823 ts_tablet_manager.cc:1398] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.054805 18833 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:39.055132 17772 tablet_service.cc:1467] Processing CreateTablet for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.055346 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.055696 18833 tablet_bootstrap.cc:654] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.056594 18833 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:39.056667 18833 ts_tablet_manager.cc:1367] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.057099 18833 raft_consensus.cc:348] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.057219 18833 raft_consensus.cc:374] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.057253 18833 raft_consensus.cc:729] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.057354 18817 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:39.057359 18833 consensus_queue.cc:260] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.057613 18833 ts_tablet_manager.cc:1398] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.057765 18833 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:39.058557 18817 tablet_bootstrap.cc:654] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.058704 18824 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:39.058758 18833 tablet_bootstrap.cc:654] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.059587 18817 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:39.059677 18817 ts_tablet_manager.cc:1367] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:39.059722 18833 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:39.059787 18833 ts_tablet_manager.cc:1367] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.060056 18818 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:39.060142 18818 ts_tablet_manager.cc:1367] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 08:00:39.060101 18817 raft_consensus.cc:348] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.060153 18833 raft_consensus.cc:348] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.060221 18817 raft_consensus.cc:374] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.060250 18817 raft_consensus.cc:729] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.060254 18833 raft_consensus.cc:374] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.060284 18833 raft_consensus.cc:729] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.060369 18817 consensus_queue.cc:260] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.060412 18833 consensus_queue.cc:260] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.060647 18817 ts_tablet_manager.cc:1398] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.060566 18818 raft_consensus.cc:348] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.060700 18818 raft_consensus.cc:374] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.060734 18818 raft_consensus.cc:729] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.060739 18833 ts_tablet_manager.cc:1398] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.060801 18817 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:39.060870 18833 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:39.060848 18818 consensus_queue.cc:260] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.061136 18818 ts_tablet_manager.cc:1398] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.059609 18824 tablet_bootstrap.cc:654] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.061625 18833 tablet_bootstrap.cc:654] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.061625 18817 tablet_bootstrap.cc:654] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.062304 18782 raft_consensus.cc:482] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:39.062397 18782 raft_consensus.cc:504] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.062561 18817 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:39.062646 18817 ts_tablet_manager.cc:1367] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.062705 18833 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:39.062781 18833 ts_tablet_manager.cc:1367] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.063040 18817 raft_consensus.cc:348] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.063093 18782 leader_election.cc:288] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:39.063149 18817 raft_consensus.cc:374] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.063179 18817 raft_consensus.cc:729] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.063144 18833 raft_consensus.cc:348] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.063236 18833 raft_consensus.cc:374] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.063263 18833 raft_consensus.cc:729] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.063285 18817 consensus_queue.cc:260] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.063369 18833 consensus_queue.cc:260] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.063539 18817 ts_tablet_manager.cc:1398] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.063598 18833 ts_tablet_manager.cc:1398] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.063684 17923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:39.063817 17923 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.064061 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:39.064177 18058 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.064404 18824 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:39.064527 18824 ts_tablet_manager.cc:1367] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:00:39.064544 18826 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:39.064628 18826 ts_tablet_manager.cc:1367] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.000s
I20240503 08:00:39.064715 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:39.064805 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:39.065095 18826 raft_consensus.cc:348] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.065218 18826 raft_consensus.cc:374] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.065260 18826 raft_consensus.cc:729] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.065233 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:39.065333 17266 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
W20240503 08:00:39.065325 17463 leader_election.cc:341] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:39.065369 18826 consensus_queue.cc:260] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.065632 18826 ts_tablet_manager.cc:1398] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.065780 18824 raft_consensus.cc:348] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.065943 18824 raft_consensus.cc:374] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.065995 18826 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:39.066015 18824 raft_consensus.cc:729] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.066229 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:39.066344 17794 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.066243 18824 consensus_queue.cc:260] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.066663 18824 ts_tablet_manager.cc:1398] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.066869 18824 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:39.066969 17463 leader_election.cc:302] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters: 0ce150b975ff4040bedde6319d5cf3d5
W20240503 08:00:39.067143 17463 leader_election.cc:341] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:39.067189 18826 tablet_bootstrap.cc:654] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.067821 18824 tablet_bootstrap.cc:654] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.068125 18826 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:39.068212 18826 ts_tablet_manager.cc:1367] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.068210 18782 raft_consensus.cc:2781] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:39.068260 18782 raft_consensus.cc:482] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:39.068306 18782 raft_consensus.cc:3037] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.068776 18826 raft_consensus.cc:348] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.068867 18826 raft_consensus.cc:374] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.068902 18826 raft_consensus.cc:729] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.069005 18826 consensus_queue.cc:260] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.069288 18826 ts_tablet_manager.cc:1398] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.069303 18782 raft_consensus.cc:504] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.069460 18826 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:39.069778 18824 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:39.069850 18824 ts_tablet_manager.cc:1367] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:39.069850 18782 leader_election.cc:288] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 election: Requested vote from peers 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:39.070237 18826 tablet_bootstrap.cc:654] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.070235 18824 raft_consensus.cc:348] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.070355 18824 raft_consensus.cc:374] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.070389 18824 raft_consensus.cc:729] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.070384 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:39.070472 17794 raft_consensus.cc:3037] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.070497 18824 consensus_queue.cc:260] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.070746 18824 ts_tablet_manager.cc:1398] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.070883 18824 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:39.071121 18826 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:39.071188 18826 ts_tablet_manager.cc:1367] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.071362 17794 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.071552 18826 raft_consensus.cc:348] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.071647 18824 tablet_bootstrap.cc:654] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.071652 18826 raft_consensus.cc:374] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.071681 18826 raft_consensus.cc:729] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.071801 18826 consensus_queue.cc:260] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.072068 18826 ts_tablet_manager.cc:1398] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.072175 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:39.072284 18058 raft_consensus.cc:3037] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.072587 18824 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:39.072651 18824 ts_tablet_manager.cc:1367] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.072966 18058 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.072957 18824 raft_consensus.cc:348] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.073029 18824 raft_consensus.cc:374] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.073048 18824 raft_consensus.cc:729] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.073199 18824 consensus_queue.cc:260] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.073314 17923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:39.073421 17923 raft_consensus.cc:3037] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.073457 18824 ts_tablet_manager.cc:1398] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:39.074056 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:39.074162 17662 raft_consensus.cc:3037] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.074158 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:39.074251 17266 raft_consensus.cc:3037] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.074339 17923 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.074733 17464 leader_election.cc:302] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:39.074894 18782 raft_consensus.cc:2781] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:39.074965 18782 raft_consensus.cc:686] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Becoming Leader. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Running, Role: LEADER
I20240503 08:00:39.074985 17662 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.075045 17266 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.075086 18782 consensus_queue.cc:237] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.075193 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a79ddd7d381c4ccbbd1a1e7d3ec8366f" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:39.075289 17397 raft_consensus.cc:3037] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.076216 17397 raft_consensus.cc:2454] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.077193 17115 catalog_manager.cc:5484] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7af8f2a481294a2cb93e5761947978cc (127.0.99.131). New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:39.103014 18782 raft_consensus.cc:482] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:39.103129 18782 raft_consensus.cc:504] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.103641 18782 leader_election.cc:288] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:39.103983 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:39.104036 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:39.104046 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:39.104130 17662 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.104163 18058 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.104163 17794 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.104285 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:39.104406 17266 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.104575 17923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:39.104588 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:39.104681 17463 leader_election.cc:302] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:39.104715 17923 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.104714 17397 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.104872 18782 raft_consensus.cc:2781] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:39.104969 18782 raft_consensus.cc:482] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:39.105033 18782 raft_consensus.cc:3037] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.105875 18782 raft_consensus.cc:504] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.106432 18782 leader_election.cc:288] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 election: Requested vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:39.106623 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:39.106640 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:39.106720 17397 raft_consensus.cc:3037] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.106734 17794 raft_consensus.cc:3037] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.106732 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:39.106817 17662 raft_consensus.cc:3037] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.106905 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:39.107010 17266 raft_consensus.cc:3037] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.107594 17662 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.107594 17397 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.107638 17794 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.107895 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:39.107980 17465 leader_election.cc:302] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:39.108017 18058 raft_consensus.cc:3037] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.108058 17923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143109561ad549d69a59992c02fe23af" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:39.108146 18782 raft_consensus.cc:2781] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:39.108168 17923 raft_consensus.cc:3037] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.108227 18782 raft_consensus.cc:686] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Becoming Leader. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Running, Role: LEADER
I20240503 08:00:39.108364 18782 consensus_queue.cc:237] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.108894 18058 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.108989 17923 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.109369 17266 raft_consensus.cc:2454] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.110549 17117 catalog_manager.cc:5484] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7af8f2a481294a2cb93e5761947978cc (127.0.99.131). New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:39.179908 18788 raft_consensus.cc:482] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:39.180033 18788 raft_consensus.cc:504] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.180655 18788 leader_election.cc:288] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:39.180960 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:39.181003 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:39.181098 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:39.181172 17662 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:39.181176 17923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:39.181221 17528 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:39.181322 17923 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:39.181358 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:39.181468 18058 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:39.181608 17200 leader_election.cc:302] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, 8eb6822290b046f88c861159aa9421ec; no voters:
I20240503 08:00:39.181654 17794 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:39.181773 18788 raft_consensus.cc:2781] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:39.181833 18788 raft_consensus.cc:482] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:39.181869 18788 raft_consensus.cc:3037] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.181900 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:39.182022 17397 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:39.182755 18788 raft_consensus.cc:504] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.183218 18788 leader_election.cc:288] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 election: Requested vote from peers f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:39.183388 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:39.183395 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:39.183480 17397 raft_consensus.cc:3037] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.183480 17794 raft_consensus.cc:3037] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.183518 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:39.183614 17528 raft_consensus.cc:3037] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.183887 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:39.183979 17662 raft_consensus.cc:3037] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.184340 17794 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:39.184334 17397 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:39.184530 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:39.184558 17528 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:39.184620 18058 raft_consensus.cc:3037] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.184852 17923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00708b0b8f2f4366944b244a9ba8c29f" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:39.184962 17923 raft_consensus.cc:3037] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.185382 18058 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:39.185611 17200 leader_election.cc:302] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 7af8f2a481294a2cb93e5761947978cc, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:39.185735 17923 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:39.185773 18788 raft_consensus.cc:2781] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:39.185806 17662 raft_consensus.cc:2454] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:39.185849 18788 raft_consensus.cc:686] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Becoming Leader. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Running, Role: LEADER
I20240503 08:00:39.186014 18788 consensus_queue.cc:237] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.188174 17117 catalog_manager.cc:5484] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eb6822290b046f88c861159aa9421ec (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:39.318213 18847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:39.318415 18847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:39.321424 18847 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 08:00:39.391543 17794 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.391921 18058 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.392313 17397 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.392366 18789 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.392634 17528 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.393051 17662 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.394042 17925 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.394537 18749 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.395249 17925 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.395553 17397 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.395915 18783 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.397215 17923 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.398106 18057 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.398394 18056 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.398658 17264 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.398959 17398 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.399037 18556 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.399473 17266 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.399572 17661 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.400581 18787 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.400701 18789 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.400846 18537 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.400951 18789 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.401476 18556 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.401518 18782 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.401959 17793 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.399549 17662 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.401959 17794 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:39.402299 18556 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.402343 18782 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.404115 18556 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.410130 18783 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.410400 18783 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.410607 18783 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.414708 18537 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:39.420167 18782 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 170.921 ms
time per row: 0.0854604 ms
I20240503 08:00:39.534883 17122 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47184:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:39.535694 17122 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_3 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:00:39.541611 17905 tablet_service.cc:1467] Processing CreateTablet for tablet ecc1f8b69268429f89cffb380e45e020 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.541918 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc1f8b69268429f89cffb380e45e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.543915 17640 tablet_service.cc:1467] Processing CreateTablet for tablet ecc1f8b69268429f89cffb380e45e020 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.544149 18527 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:39.544205 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc1f8b69268429f89cffb380e45e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.544236 17246 tablet_service.cc:1467] Processing CreateTablet for tablet ecc1f8b69268429f89cffb380e45e020 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.544456 17246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc1f8b69268429f89cffb380e45e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.544588 17244 tablet_service.cc:1467] Processing CreateTablet for tablet ed32a1099e044448aaa0c9c9f58f74e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.544791 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32a1099e044448aaa0c9c9f58f74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.544924 18527 tablet_bootstrap.cc:654] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.546411 18824 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:39.546414 18817 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:39.546782 17245 tablet_service.cc:1467] Processing CreateTablet for tablet 10e598a3a124472eab35c97689057ace (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.546981 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.547134 18817 tablet_bootstrap.cc:654] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.547214 18824 tablet_bootstrap.cc:654] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.548903 17378 tablet_service.cc:1467] Processing CreateTablet for tablet ecc1f8b69268429f89cffb380e45e020 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.549419 17377 tablet_service.cc:1467] Processing CreateTablet for tablet ed32a1099e044448aaa0c9c9f58f74e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.549633 17377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32a1099e044448aaa0c9c9f58f74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.551895 17376 tablet_service.cc:1467] Processing CreateTablet for tablet 10e598a3a124472eab35c97689057ace (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.552114 17376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.554276 18826 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:39.554385 18817 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:39.554464 18817 ts_tablet_manager.cc:1367] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:00:39.554965 18817 raft_consensus.cc:348] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.555099 18826 tablet_bootstrap.cc:654] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.555117 18817 raft_consensus.cc:374] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.555151 18817 raft_consensus.cc:729] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.555284 18817 consensus_queue.cc:260] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.555585 18817 ts_tablet_manager.cc:1398] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.555774 18817 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:39.556658 18817 tablet_bootstrap.cc:654] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.557019 18038 tablet_service.cc:1467] Processing CreateTablet for tablet ed32a1099e044448aaa0c9c9f58f74e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.557322 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32a1099e044448aaa0c9c9f58f74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.558238 18817 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:39.558262 18037 tablet_service.cc:1467] Processing CreateTablet for tablet ecc1f8b69268429f89cffb380e45e020 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.558323 18817 ts_tablet_manager.cc:1367] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:39.558483 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc1f8b69268429f89cffb380e45e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.558756 18817 raft_consensus.cc:348] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.558867 18817 raft_consensus.cc:374] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.558897 18817 raft_consensus.cc:729] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.559026 18817 consensus_queue.cc:260] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.559310 18817 ts_tablet_manager.cc:1398] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.559468 18817 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:39.559490 17905 tablet_service.cc:1467] Processing CreateTablet for tablet ed32a1099e044448aaa0c9c9f58f74e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.559736 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32a1099e044448aaa0c9c9f58f74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.560356 18817 tablet_bootstrap.cc:654] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.560565 18036 tablet_service.cc:1467] Processing CreateTablet for tablet 10e598a3a124472eab35c97689057ace (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.560775 18036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.561210 18818 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:39.561981 18818 tablet_bootstrap.cc:654] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.562601 17904 tablet_service.cc:1467] Processing CreateTablet for tablet 10e598a3a124472eab35c97689057ace (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.562853 17904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.563441 18818 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:39.563519 18818 ts_tablet_manager.cc:1367] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.563666 17642 tablet_service.cc:1467] Processing CreateTablet for tablet 10e598a3a124472eab35c97689057ace (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.563953 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.563663 17640 tablet_service.cc:1467] Processing CreateTablet for tablet ed32a1099e044448aaa0c9c9f58f74e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.563979 18818 raft_consensus.cc:348] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.564076 18818 raft_consensus.cc:374] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.564105 18818 raft_consensus.cc:729] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.564205 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32a1099e044448aaa0c9c9f58f74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.564210 18818 consensus_queue.cc:260] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.564836 17509 tablet_service.cc:1467] Processing CreateTablet for tablet ecc1f8b69268429f89cffb380e45e020 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.565014 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 10e598a3a124472eab35c97689057ace (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.565131 17509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc1f8b69268429f89cffb380e45e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.565248 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.565634 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc1f8b69268429f89cffb380e45e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.567092 18818 ts_tablet_manager.cc:1398] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:39.564836 17508 tablet_service.cc:1467] Processing CreateTablet for tablet ed32a1099e044448aaa0c9c9f58f74e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.567277 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32a1099e044448aaa0c9c9f58f74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.567299 18818 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:39.568194 18818 tablet_bootstrap.cc:654] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.568148 17772 tablet_service.cc:1467] Processing CreateTablet for tablet ecc1f8b69268429f89cffb380e45e020 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.568424 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc1f8b69268429f89cffb380e45e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.569507 18818 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:39.569591 18818 ts_tablet_manager.cc:1367] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:39.570093 18527 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:39.570009 18818 raft_consensus.cc:348] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.570147 18818 raft_consensus.cc:374] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.570173 18527 ts_tablet_manager.cc:1367] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 08:00:39.570185 18818 raft_consensus.cc:729] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.570305 18818 consensus_queue.cc:260] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.570609 18818 ts_tablet_manager.cc:1398] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.570590 18527 raft_consensus.cc:348] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.570696 18527 raft_consensus.cc:374] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.570729 18527 raft_consensus.cc:729] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.570778 18818 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:39.570735 17774 tablet_service.cc:1467] Processing CreateTablet for tablet ed32a1099e044448aaa0c9c9f58f74e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.570837 18527 consensus_queue.cc:260] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.570946 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32a1099e044448aaa0c9c9f58f74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.571112 18527 ts_tablet_manager.cc:1398] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.571277 18527 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:39.571833 18818 tablet_bootstrap.cc:654] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.572328 18527 tablet_bootstrap.cc:654] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.572940 17773 tablet_service.cc:1467] Processing CreateTablet for tablet 10e598a3a124472eab35c97689057ace (DEFAULT_TABLE table=rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:39.573186 17773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:39.573410 18527 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:39.573482 18527 ts_tablet_manager.cc:1367] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:39.573851 18527 raft_consensus.cc:348] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.573971 18527 raft_consensus.cc:374] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.574007 18527 raft_consensus.cc:729] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.574138 18527 consensus_queue.cc:260] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.574457 18527 ts_tablet_manager.cc:1398] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.574609 18527 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:39.575384 18527 tablet_bootstrap.cc:654] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.577499 18745 raft_consensus.cc:482] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:39.577597 18745 raft_consensus.cc:504] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.578074 18910 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:39.578333 18745 leader_election.cc:288] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:39.578485 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:39.578928 18912 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:39.578894 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:39.579016 17924 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
W20240503 08:00:39.579005 17201 leader_election.cc:341] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:39.579209 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:39.579331 18910 tablet_bootstrap.cc:654] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.579322 18057 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:39.579790 18912 tablet_bootstrap.cc:654] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.580307 18824 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:39.580273 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:39.580395 18824 ts_tablet_manager.cc:1367] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.001s
I20240503 08:00:39.580528 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:39.580708 18826 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:39.580780 18826 ts_tablet_manager.cc:1367] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.000s
I20240503 08:00:39.580803 18824 raft_consensus.cc:348] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.580915 18824 raft_consensus.cc:374] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.580946 18824 raft_consensus.cc:729] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
W20240503 08:00:39.581075 17199 leader_election.cc:341] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:39.581075 18824 consensus_queue.cc:260] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.581113 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
W20240503 08:00:39.581174 17199 leader_election.cc:341] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:39.581213 18826 raft_consensus.cc:348] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.581333 18826 raft_consensus.cc:374] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.581365 18826 raft_consensus.cc:729] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.581373 18824 ts_tablet_manager.cc:1398] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 08:00:39.581476 17199 leader_election.cc:341] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:39.581516 18824 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:39.581477 18826 consensus_queue.cc:260] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.581529 17199 leader_election.cc:302] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters: 0ce150b975ff4040bedde6319d5cf3d5, 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, f0a940e3a6e340a69ec7ac31d26c9298
I20240503 08:00:39.581741 18826 ts_tablet_manager.cc:1398] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.581758 18745 raft_consensus.cc:2726] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:00:39.581892 18826 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:39.582509 18824 tablet_bootstrap.cc:654] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.582654 18826 tablet_bootstrap.cc:654] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.583447 18824 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:39.583514 18824 ts_tablet_manager.cc:1367] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:39.583781 18910 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:39.583909 18910 ts_tablet_manager.cc:1367] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:00:39.583866 18824 raft_consensus.cc:348] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.583978 18824 raft_consensus.cc:374] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.584012 18824 raft_consensus.cc:729] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.584139 18824 consensus_queue.cc:260] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.584404 18824 ts_tablet_manager.cc:1398] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.584586 18824 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:39.584583 18910 raft_consensus.cc:348] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.584781 18910 raft_consensus.cc:374] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.584853 18910 raft_consensus.cc:729] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.585107 18910 consensus_queue.cc:260] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.585354 18818 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:39.585436 18818 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 08:00:39.585824 18818 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.585940 18818 raft_consensus.cc:374] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.585973 18818 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.585918 18910 ts_tablet_manager.cc:1398] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.586086 18818 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.586392 18818 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.586588 18910 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:39.586925 18824 tablet_bootstrap.cc:654] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.589288 18910 tablet_bootstrap.cc:654] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.589681 18826 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:39.589769 18826 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:00:39.590199 18826 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.590289 18826 raft_consensus.cc:374] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.590308 18826 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.590415 18826 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.590510 18910 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:39.590586 18910 ts_tablet_manager.cc:1367] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:39.590662 18826 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.590781 18826 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:39.590955 18910 raft_consensus.cc:348] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.591058 18910 raft_consensus.cc:374] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.591090 18910 raft_consensus.cc:729] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.591475 18826 tablet_bootstrap.cc:654] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.591703 18910 consensus_queue.cc:260] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.592451 18912 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:39.592543 18912 ts_tablet_manager.cc:1367] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 08:00:39.592887 18910 ts_tablet_manager.cc:1398] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:39.592947 18912 raft_consensus.cc:348] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.593086 18912 raft_consensus.cc:374] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.593096 18910 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:39.593120 18912 raft_consensus.cc:729] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.593231 18912 consensus_queue.cc:260] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.593500 18912 ts_tablet_manager.cc:1398] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:39.593668 18912 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:39.594475 18912 tablet_bootstrap.cc:654] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.594940 18910 tablet_bootstrap.cc:654] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.595402 18912 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:39.595474 18912 ts_tablet_manager.cc:1367] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:39.595826 18912 raft_consensus.cc:348] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.595918 18912 raft_consensus.cc:374] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.595947 18912 raft_consensus.cc:729] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.596081 18912 consensus_queue.cc:260] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.596321 18912 ts_tablet_manager.cc:1398] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.596477 18912 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:39.597318 18912 tablet_bootstrap.cc:654] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:39.597364 18527 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:39.597451 18527 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.000s
I20240503 08:00:39.597851 18527 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.597965 18527 raft_consensus.cc:374] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.598007 18527 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.598134 18527 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.598430 18527 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.598842 18817 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:39.598942 18817 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20240503 08:00:39.598976 18912 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:39.599056 18912 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:39.599368 18817 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.599428 18912 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.599490 18817 raft_consensus.cc:374] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.599517 18912 raft_consensus.cc:374] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.599534 18817 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.599545 18912 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.599639 18817 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.599656 18912 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.599887 18912 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.599903 18817 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:39.605381 18824 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:39.605490 18824 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 08:00:39.605829 18824 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.605947 18824 raft_consensus.cc:374] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.605978 18824 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.606093 18824 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.606336 18824 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.613395 18910 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:39.613493 18910 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 08:00:39.613868 18910 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.613967 18910 raft_consensus.cc:374] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.613996 18910 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.614113 18910 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.614362 18910 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.617406 18826 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:39.617507 18826 ts_tablet_manager.cc:1367] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.000s
I20240503 08:00:39.617894 18826 raft_consensus.cc:348] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.617996 18826 raft_consensus.cc:374] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:39.618028 18826 raft_consensus.cc:729] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:39.618144 18826 consensus_queue.cc:260] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.618393 18826 ts_tablet_manager.cc:1398] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:39.625417 18783 raft_consensus.cc:482] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:39.625536 18783 raft_consensus.cc:504] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.626153 18783 leader_election.cc:288] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:39.626322 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:39.626322 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:39.626439 17660 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.626439 17792 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.626658 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:39.626760 18057 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.626897 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:39.627033 17266 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.627074 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:39.627168 17397 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.627163 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:39.627286 17924 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 0.
I20240503 08:00:39.627516 17464 leader_election.cc:302] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:39.627722 18783 raft_consensus.cc:2781] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:39.627768 18783 raft_consensus.cc:482] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:39.627799 18783 raft_consensus.cc:3037] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.628751 18783 raft_consensus.cc:504] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.629990 18783 leader_election.cc:288] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 election: Requested vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:39.630125 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:39.630223 17792 raft_consensus.cc:3037] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.630376 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:39.630476 18057 raft_consensus.cc:3037] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.631183 17792 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.631376 18057 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.631832 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:39.631856 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:39.631934 17660 raft_consensus.cc:3037] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.631934 17397 raft_consensus.cc:3037] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.631908 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:39.632006 17266 raft_consensus.cc:3037] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.632781 17397 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.632781 17660 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.632818 17266 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.633136 17463 leader_election.cc:302] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:39.633411 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e598a3a124472eab35c97689057ace" candidate_uuid: "7af8f2a481294a2cb93e5761947978cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:39.633528 17924 raft_consensus.cc:3037] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.634406 17924 raft_consensus.cc:2454] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af8f2a481294a2cb93e5761947978cc in term 1.
I20240503 08:00:39.634711 18783 raft_consensus.cc:2781] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:39.634785 18783 raft_consensus.cc:686] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Becoming Leader. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Running, Role: LEADER
I20240503 08:00:39.634923 18783 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.636961 17116 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7af8f2a481294a2cb93e5761947978cc (127.0.99.131). New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:39.651270 18661 raft_consensus.cc:482] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:39.651372 18661 raft_consensus.cc:504] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.651916 18661 leader_election.cc:288] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:39.652207 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:39.652271 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:39.652331 17792 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:39.652376 17660 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:39.652649 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:39.652770 17528 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:39.652762 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:39.652814 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:39.652880 17266 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:39.652915 18057 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:39.653115 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:39.653203 17331 leader_election.cc:302] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:39.653244 17924 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:39.653429 18661 raft_consensus.cc:2781] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:39.653481 18661 raft_consensus.cc:482] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:39.653517 18661 raft_consensus.cc:3037] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.654294 18661 raft_consensus.cc:504] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.654814 18661 leader_election.cc:288] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:39.655076 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:39.655167 17660 raft_consensus.cc:3037] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.655155 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:39.655160 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:39.655238 17924 raft_consensus.cc:3037] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.655248 17528 raft_consensus.cc:3037] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.656025 17660 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:39.656029 17924 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:39.656174 17528 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:39.656474 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:39.656497 17331 leader_election.cc:302] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:39.656574 17792 raft_consensus.cc:3037] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.656630 18661 raft_consensus.cc:2781] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:39.656692 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:39.656780 18057 raft_consensus.cc:3037] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.656792 18661 raft_consensus.cc:686] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Becoming Leader. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Running, Role: LEADER
I20240503 08:00:39.656929 18661 consensus_queue.cc:237] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:39.657490 17792 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:39.657600 18057 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:39.658313 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32a1099e044448aaa0c9c9f58f74e6" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:39.658416 17266 raft_consensus.cc:3037] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.658979 17116 catalog_manager.cc:5484] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:39.659242 17266 raft_consensus.cc:2454] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:39.809772 18797 raft_consensus.cc:482] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:39.809907 18797 raft_consensus.cc:504] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.810536 18797 leader_election.cc:288] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:39.811003 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:39.810982 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:39.811008 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:39.811142 17397 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:39.811142 17924 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:39.811146 17792 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:39.811573 17596 leader_election.cc:302] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:39.811589 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:39.811605 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:39.811678 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:39.811713 17266 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:39.811726 17528 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:39.811745 18797 raft_consensus.cc:2781] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:39.811786 18797 raft_consensus.cc:482] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:39.811786 18057 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:39.811820 18797 raft_consensus.cc:3037] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.812741 18797 raft_consensus.cc:504] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.813252 18797 leader_election.cc:288] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 election: Requested vote from peers 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:39.813527 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:39.813625 18057 raft_consensus.cc:3037] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.813725 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:39.813804 17924 raft_consensus.cc:3037] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.814400 18057 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:39.814594 17924 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:39.814992 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:39.815099 17792 raft_consensus.cc:3037] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.815716 17792 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:39.815990 17597 leader_election.cc:302] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:39.816148 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:39.816248 17266 raft_consensus.cc:3037] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.816316 18797 raft_consensus.cc:2781] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:39.816354 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:39.816457 17528 raft_consensus.cc:3037] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.816475 18797 raft_consensus.cc:686] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Running, Role: LEADER
I20240503 08:00:39.816608 18797 consensus_queue.cc:237] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:39.816931 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc1f8b69268429f89cffb380e45e020" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:39.817035 17397 raft_consensus.cc:3037] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:39.817159 17266 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:39.817350 17528 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:39.817852 17397 raft_consensus.cc:2454] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:39.818989 17116 catalog_manager.cc:5484] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:39.974633 18931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:39.974913 18931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:39.978099 18931 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 08:00:40.025256 18783 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.037570 18661 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.063701 18872 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.067118 17266 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.067395 18057 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.067459 17397 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.067852 18797 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.068118 17924 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.068838 17794 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.069298 18929 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.069415 17528 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.069597 18929 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.069820 18929 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.070442 18929 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.074307 17923 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.074679 18661 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.075726 17792 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.076079 18872 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.077983 17266 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.079867 17793 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.080211 18058 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.080472 18872 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.080488 18927 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.080826 18929 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.081847 18927 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.082470 17397 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.082760 17266 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.082934 18872 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.083943 18661 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.084549 17925 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.085789 18783 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.089089 17529 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.089506 18661 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.129765 18995 mvcc.cc:205] Tried to move back new op lower bound from 7023506391451705344 to 7023506389637804032. Current Snapshot: MvccSnapshot[applied={T|T < 7023506391339249664}]
I20240503 08:00:40.140208 19000 mvcc.cc:205] Tried to move back new op lower bound from 7023506391339249664 to 7023506389637804032. Current Snapshot: MvccSnapshot[applied={T|T < 7023506391339249664}]
I20240503 08:00:40.150295 18978 mvcc.cc:205] Tried to move back new op lower bound from 7023506391328251904 to 7023506390292160512. Current Snapshot: MvccSnapshot[applied={T|T < 7023506391311474688}]
I20240503 08:00:40.182870 18975 mvcc.cc:205] Tried to move back new op lower bound from 7023506391367364608 to 7023506389547708416. Current Snapshot: MvccSnapshot[applied={T|T < 7023506391341690880}]
I20240503 08:00:40.193842 18977 mvcc.cc:205] Tried to move back new op lower bound from 7023506391367364608 to 7023506389547708416. Current Snapshot: MvccSnapshot[applied={T|T < 7023506391367364608}]
I20240503 08:00:40.212657 18984 mvcc.cc:205] Tried to move back new op lower bound from 7023506391367364608 to 7023506389547708416. Current Snapshot: MvccSnapshot[applied={T|T < 7023506391341690880}]
INSERT report
rows total: 2000
time total: 208.899 ms
time per row: 0.10445 ms
I20240503 08:00:40.226837 17121 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47184:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:40.227317 17121 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_4 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:00:40.231770 17245 tablet_service.cc:1467] Processing CreateTablet for tablet 99d8d89b8561420781b7e7a9b9c778a0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.231829 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 99d8d89b8561420781b7e7a9b9c778a0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.231854 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 99d8d89b8561420781b7e7a9b9c778a0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.232059 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d8d89b8561420781b7e7a9b9c778a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.232059 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d8d89b8561420781b7e7a9b9c778a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.232112 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d8d89b8561420781b7e7a9b9c778a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.234144 17904 tablet_service.cc:1467] Processing CreateTablet for tablet 99d8d89b8561420781b7e7a9b9c778a0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.234393 17904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d8d89b8561420781b7e7a9b9c778a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.234578 18036 tablet_service.cc:1467] Processing CreateTablet for tablet 99d8d89b8561420781b7e7a9b9c778a0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.234668 18037 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1de048a83e45548afac96309abf4d2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.234820 18036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d8d89b8561420781b7e7a9b9c778a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.235221 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1de048a83e45548afac96309abf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.235384 17642 tablet_service.cc:1467] Processing CreateTablet for tablet 99d8d89b8561420781b7e7a9b9c778a0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.235620 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d8d89b8561420781b7e7a9b9c778a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.236593 17905 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1de048a83e45548afac96309abf4d2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.236797 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1de048a83e45548afac96309abf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.236982 18038 tablet_service.cc:1467] Processing CreateTablet for tablet ab988cc96fc940239e18f54ec9c9b401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.237214 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.237305 17640 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1de048a83e45548afac96309abf4d2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.237493 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1de048a83e45548afac96309abf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.237798 17773 tablet_service.cc:1467] Processing CreateTablet for tablet 99d8d89b8561420781b7e7a9b9c778a0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.238029 17773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d8d89b8561420781b7e7a9b9c778a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.238652 17906 tablet_service.cc:1467] Processing CreateTablet for tablet ab988cc96fc940239e18f54ec9c9b401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.238840 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.239300 17244 tablet_service.cc:1467] Processing CreateTablet for tablet ab988cc96fc940239e18f54ec9c9b401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.239523 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.239826 17508 tablet_service.cc:1467] Processing CreateTablet for tablet ab988cc96fc940239e18f54ec9c9b401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.240026 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.240131 17774 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1de048a83e45548afac96309abf4d2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.240339 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1de048a83e45548afac96309abf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.240706 17641 tablet_service.cc:1467] Processing CreateTablet for tablet ab988cc96fc940239e18f54ec9c9b401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.240898 17641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.241323 17772 tablet_service.cc:1467] Processing CreateTablet for tablet ab988cc96fc940239e18f54ec9c9b401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.241539 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.241829 17245 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1de048a83e45548afac96309abf4d2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.242028 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1de048a83e45548afac96309abf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.242837 17376 tablet_service.cc:1467] Processing CreateTablet for tablet ab988cc96fc940239e18f54ec9c9b401 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.243058 17376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.243196 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1de048a83e45548afac96309abf4d2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.243403 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1de048a83e45548afac96309abf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.244390 19040 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:40.244967 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1de048a83e45548afac96309abf4d2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.245199 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1de048a83e45548afac96309abf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.245519 19040 tablet_bootstrap.cc:654] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.245565 19041 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:40.245805 19039 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:40.246371 19041 tablet_bootstrap.cc:654] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.246560 19039 tablet_bootstrap.cc:654] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.247159 19036 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:40.247246 19040 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:40.247326 19040 ts_tablet_manager.cc:1367] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:40.247344 19042 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:40.247548 19039 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:40.247630 19039 ts_tablet_manager.cc:1367] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:40.247761 19040 raft_consensus.cc:348] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.247881 19040 raft_consensus.cc:374] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.247917 19040 raft_consensus.cc:729] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.248095 19036 tablet_bootstrap.cc:654] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.248026 19040 consensus_queue.cc:260] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.248019 19039 raft_consensus.cc:348] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.248142 19039 raft_consensus.cc:374] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.248179 19039 raft_consensus.cc:729] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.248317 19040 ts_tablet_manager.cc:1398] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.248282 19039 consensus_queue.cc:260] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.248482 19040 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:40.248531 19039 ts_tablet_manager.cc:1398] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.248688 19039 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:40.248952 19042 tablet_bootstrap.cc:654] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.248988 19041 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:40.249092 19041 ts_tablet_manager.cc:1367] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:00:40.248711 19038 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:40.249285 19040 tablet_bootstrap.cc:654] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.249467 19039 tablet_bootstrap.cc:654] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.249509 19041 raft_consensus.cc:348] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.249629 19041 raft_consensus.cc:374] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.249665 19041 raft_consensus.cc:729] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.249812 19041 consensus_queue.cc:260] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.250157 19041 ts_tablet_manager.cc:1398] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.250383 19037 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:40.251139 19037 tablet_bootstrap.cc:654] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.251636 19039 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:40.251708 19039 ts_tablet_manager.cc:1367] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:40.252027 19036 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:40.252110 19036 ts_tablet_manager.cc:1367] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:00:40.252063 19039 raft_consensus.cc:348] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.252194 19039 raft_consensus.cc:374] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.252228 19039 raft_consensus.cc:729] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.252331 19039 consensus_queue.cc:260] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.252584 19039 ts_tablet_manager.cc:1398] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.252524 19036 raft_consensus.cc:348] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.252631 19036 raft_consensus.cc:374] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.252662 19036 raft_consensus.cc:729] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.252732 19039 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:40.252799 19036 consensus_queue.cc:260] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.253198 19036 ts_tablet_manager.cc:1398] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.253371 19036 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:40.253702 19041 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:40.254135 19036 tablet_bootstrap.cc:654] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.254647 19041 tablet_bootstrap.cc:654] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.254850 19040 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:40.254952 19040 ts_tablet_manager.cc:1367] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:00:40.255143 19036 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:40.255219 19036 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:40.255365 19040 raft_consensus.cc:348] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.255492 19040 raft_consensus.cc:374] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.255528 19040 raft_consensus.cc:729] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.255594 19036 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.255637 19040 consensus_queue.cc:260] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.255699 19036 raft_consensus.cc:374] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.255728 19036 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.255882 19040 ts_tablet_manager.cc:1398] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.255879 19036 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.256032 19040 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:40.256095 19039 tablet_bootstrap.cc:654] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.256186 19036 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.256342 19036 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:40.256655 19038 tablet_bootstrap.cc:654] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.256822 19040 tablet_bootstrap.cc:654] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.257432 19036 tablet_bootstrap.cc:654] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.258523 19037 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:40.258608 19037 ts_tablet_manager.cc:1367] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:00:40.258708 19038 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:40.258795 19038 ts_tablet_manager.cc:1367] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:00:40.258991 19037 raft_consensus.cc:348] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.259112 19037 raft_consensus.cc:374] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.259145 19037 raft_consensus.cc:729] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.259183 19038 raft_consensus.cc:348] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.259300 19038 raft_consensus.cc:374] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.259327 19038 raft_consensus.cc:729] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.259274 19037 consensus_queue.cc:260] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.259452 19038 consensus_queue.cc:260] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.259774 19038 ts_tablet_manager.cc:1398] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.259930 19038 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:40.260572 19038 tablet_bootstrap.cc:654] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.261327 19040 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:40.261425 19040 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:00:40.261644 19037 ts_tablet_manager.cc:1398] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:40.261860 19037 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:40.261831 19040 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.261929 19040 raft_consensus.cc:374] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.261965 19040 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.262084 19040 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.262355 19040 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.262620 19037 tablet_bootstrap.cc:654] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.262840 19039 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:40.262912 19039 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240503 08:00:40.263301 19039 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.263414 19039 raft_consensus.cc:374] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.263450 19039 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.263605 19036 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:40.263556 19039 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.263684 19036 ts_tablet_manager.cc:1367] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:00:40.263754 19037 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:40.263759 19042 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:40.263825 19039 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.263839 19042 ts_tablet_manager.cc:1367] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240503 08:00:40.264050 19036 raft_consensus.cc:348] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.264158 19036 raft_consensus.cc:374] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.264187 19036 raft_consensus.cc:729] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.264248 19042 raft_consensus.cc:348] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.264286 19036 consensus_queue.cc:260] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.264370 19042 raft_consensus.cc:374] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.264405 19042 raft_consensus.cc:729] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.264456 19037 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:40.264550 19036 ts_tablet_manager.cc:1398] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.264513 19042 consensus_queue.cc:260] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.264772 19042 ts_tablet_manager.cc:1398] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.264937 19042 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:40.264937 19041 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:40.265014 19041 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 08:00:40.265043 19037 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.265255 19037 raft_consensus.cc:374] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.265326 19037 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.265458 19041 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.265586 19041 raft_consensus.cc:374] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.265491 19037 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.265632 19041 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.265758 19042 tablet_bootstrap.cc:654] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.265748 19041 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.265970 19037 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.266028 19041 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.266191 19037 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:40.266201 19041 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:40.266961 19041 tablet_bootstrap.cc:654] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.267242 19037 tablet_bootstrap.cc:654] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.267899 19041 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:40.267974 19041 ts_tablet_manager.cc:1367] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:40.268350 19041 raft_consensus.cc:348] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.268452 19041 raft_consensus.cc:374] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.268484 19041 raft_consensus.cc:729] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.268587 19041 consensus_queue.cc:260] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.268831 19041 ts_tablet_manager.cc:1398] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.270570 19038 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:40.270658 19038 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 08:00:40.271059 19038 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.271193 19038 raft_consensus.cc:374] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.271234 19038 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.271358 19038 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.271621 19038 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:00:40.271713 19037 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:40.271780 19038 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:40.271790 19037 ts_tablet_manager.cc:1367] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:40.272161 19037 raft_consensus.cc:348] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.272276 19037 raft_consensus.cc:374] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.272310 19037 raft_consensus.cc:729] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.272431 19037 consensus_queue.cc:260] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.272558 19038 tablet_bootstrap.cc:654] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.272758 19037 ts_tablet_manager.cc:1398] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.273890 19038 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:40.273975 19038 ts_tablet_manager.cc:1367] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:00:40.274365 19038 raft_consensus.cc:348] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.274474 19038 raft_consensus.cc:374] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.274514 19038 raft_consensus.cc:729] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.274626 19038 consensus_queue.cc:260] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.274715 19017 raft_consensus.cc:482] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:40.274799 19017 raft_consensus.cc:504] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.274946 19038 ts_tablet_manager.cc:1398] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:00:40.275393 19017 leader_election.cc:288] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:40.275573 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:40.275573 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:40.275628 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:40.275674 17660 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.275664 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:40.275727 17266 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.275755 17925 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.275949 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:40.276005 17332 leader_election.cc:302] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:40.276052 18058 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.276144 19017 raft_consensus.cc:2781] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:40.276182 19017 raft_consensus.cc:482] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:40.276178 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:40.276217 19017 raft_consensus.cc:3037] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.276274 17528 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
W20240503 08:00:40.276309 17333 leader_election.cc:341] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:40.277225 19017 raft_consensus.cc:504] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.277748 19017 leader_election.cc:288] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:40.277905 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:40.278019 17266 raft_consensus.cc:3037] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.278023 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:40.278120 17660 raft_consensus.cc:3037] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.278430 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:40.278541 17925 raft_consensus.cc:3037] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.278865 17660 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.278926 17266 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.279322 17925 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.279549 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:40.279649 18058 raft_consensus.cc:3037] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.280351 18058 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.280642 17331 leader_election.cc:302] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:40.280663 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:40.280751 17528 raft_consensus.cc:3037] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.280810 19017 raft_consensus.cc:2781] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:40.280889 19017 raft_consensus.cc:686] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Becoming Leader. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Running, Role: LEADER
I20240503 08:00:40.281018 19017 consensus_queue.cc:237] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.281213 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:40.281637 17528 raft_consensus.cc:2454] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
W20240503 08:00:40.282359 17333 leader_election.cc:341] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:40.283166 17116 catalog_manager.cc:5484] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:40.287802 18815 raft_consensus.cc:482] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:40.287895 18815 raft_consensus.cc:504] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.288460 18815 leader_election.cc:288] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:40.288558 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:40.288894 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:40.288894 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:40.288995 17660 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.288995 17398 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.288982 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:40.289093 17266 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.289211 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:40.289309 18058 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.289338 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:40.289455 17528 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.289446 17859 leader_election.cc:302] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
W20240503 08:00:40.289541 17861 leader_election.cc:341] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:40.289724 18815 raft_consensus.cc:2781] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:40.289779 18815 raft_consensus.cc:482] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:40.289819 18815 raft_consensus.cc:3037] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.290640 18815 raft_consensus.cc:504] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.291123 18815 leader_election.cc:288] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 election: Requested vote from peers 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:40.291419 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:40.291419 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:40.291452 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:40.291507 17266 raft_consensus.cc:3037] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.291507 18058 raft_consensus.cc:3037] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.291568 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:40.291646 17528 raft_consensus.cc:3037] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.291671 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:40.291744 17660 raft_consensus.cc:3037] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.292255 18058 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.292255 17266 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.292407 17528 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.292477 17660 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.292711 17859 leader_election.cc:302] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:40.292740 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:40.292822 17398 raft_consensus.cc:3037] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.293347 19042 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:40.293435 19042 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 08:00:40.293524 18815 raft_consensus.cc:2781] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:40.293642 17398 raft_consensus.cc:2454] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.293679 18815 raft_consensus.cc:686] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Becoming Leader. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Running, Role: LEADER
I20240503 08:00:40.293797 18815 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
W20240503 08:00:40.293896 17861 leader_election.cc:341] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:00:40.293854 19042 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.293958 19042 raft_consensus.cc:374] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.293991 19042 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.294116 19042 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.294359 19042 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:40.294517 19042 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:40.295325 19042 tablet_bootstrap.cc:654] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.296236 19042 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:40.296319 19042 ts_tablet_manager.cc:1367] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:40.296679 19042 raft_consensus.cc:348] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.296783 19042 raft_consensus.cc:374] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.296813 19042 raft_consensus.cc:729] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.296927 19042 consensus_queue.cc:260] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.296921 17116 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:40.297197 19042 ts_tablet_manager.cc:1398] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.315320 18745 raft_consensus.cc:482] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:40.315434 18745 raft_consensus.cc:504] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.316030 18745 leader_election.cc:288] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:40.316228 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:40.316347 17398 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:40.316311 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:40.316335 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:40.316431 17660 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:40.316442 17528 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:40.316684 17199 leader_election.cc:302] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 7af8f2a481294a2cb93e5761947978cc, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:40.316900 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:40.316973 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:40.317023 17793 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:40.317085 17925 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:40.317085 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:40.317200 18058 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 0.
I20240503 08:00:40.317255 18745 raft_consensus.cc:2781] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:40.317312 18745 raft_consensus.cc:482] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:40.317348 18745 raft_consensus.cc:3037] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.318377 18745 raft_consensus.cc:504] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.318881 18745 leader_election.cc:288] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 election: Requested vote from peers f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:40.319058 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:40.319151 17398 raft_consensus.cc:3037] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.319151 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:40.319195 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:40.319252 17660 raft_consensus.cc:3037] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.319279 17925 raft_consensus.cc:3037] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.319294 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:40.319391 18058 raft_consensus.cc:3037] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.319996 17398 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:40.320089 17925 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:40.320125 17660 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:40.320145 18058 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:40.320403 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:40.320403 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d8d89b8561420781b7e7a9b9c778a0" candidate_uuid: "8eb6822290b046f88c861159aa9421ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:40.320503 17528 raft_consensus.cc:3037] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.320503 17793 raft_consensus.cc:3037] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.320597 17199 leader_election.cc:302] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:40.320742 18745 raft_consensus.cc:2781] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:40.320919 18745 raft_consensus.cc:686] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Becoming Leader. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Running, Role: LEADER
I20240503 08:00:40.321084 18745 consensus_queue.cc:237] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.321349 17793 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:40.321352 17528 raft_consensus.cc:2454] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6822290b046f88c861159aa9421ec in term 1.
I20240503 08:00:40.323081 17121 catalog_manager.cc:5484] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eb6822290b046f88c861159aa9421ec (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:40.358183 18551 raft_consensus.cc:482] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:40.358294 18551 raft_consensus.cc:504] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.359417 18551 leader_election.cc:288] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:40.366760 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:40.366919 17398 raft_consensus.cc:2379] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:40.367195 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:40.367911 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:40.368052 17660 raft_consensus.cc:2379] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:40.368990 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:40.369184 17528 raft_consensus.cc:2379] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:40.369565 17727 leader_election.cc:302] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 84305d92b082406380b107114e4cd61b; no voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, f0a940e3a6e340a69ec7ac31d26c9298
I20240503 08:00:40.369807 18551 raft_consensus.cc:3037] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.370134 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:40.370162 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab988cc96fc940239e18f54ec9c9b401" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:40.370278 17266 raft_consensus.cc:2379] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:40.370278 18058 raft_consensus.cc:2379] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate 78e52cf3c8fa435bb5ca36061227ac41 in this term.
I20240503 08:00:40.370918 18551 raft_consensus.cc:2726] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:00:40.420598 18551 raft_consensus.cc:482] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:40.420723 18551 raft_consensus.cc:504] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.421411 18551 leader_election.cc:288] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:40.421552 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:40.421705 17266 raft_consensus.cc:2379] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:40.421941 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:40.422021 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:40.422091 17925 raft_consensus.cc:2379] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:40.422139 17660 raft_consensus.cc:2379] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:40.422542 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:40.422550 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:40.422678 17528 raft_consensus.cc:2379] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:40.422873 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1de048a83e45548afac96309abf4d2" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:40.423017 18058 raft_consensus.cc:2379] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84305d92b082406380b107114e4cd61b in current term 1: Already voted for candidate f0a940e3a6e340a69ec7ac31d26c9298 in this term.
I20240503 08:00:40.423280 17727 leader_election.cc:302] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 84305d92b082406380b107114e4cd61b; no voters: 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298
I20240503 08:00:40.423436 18551 raft_consensus.cc:3037] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.424279 18551 raft_consensus.cc:2726] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 08:00:40.437418 19070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:40.437601 19070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:40.440477 19070 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:00:40.527643 17793 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.528064 18815 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.528234 17266 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.528650 18058 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.528940 17398 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.528973 17528 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.529879 17661 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.530134 17660 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.531335 18745 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.531883 17794 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.532064 19105 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.532347 19105 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.532377 18927 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.532613 19105 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.532714 17925 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.532821 19105 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.533816 18927 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.535245 18058 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.535773 17660 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.536192 19065 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.537542 19017 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.538058 17266 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.538754 17528 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.539784 18058 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.540113 18745 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.541217 19016 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.541510 19016 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.542711 17924 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.544591 17528 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.544940 19068 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.546037 17397 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.546437 18745 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.552111 19016 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.556573 19068 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.564823 17791 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:40.565439 19068 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:40.585469 18964 mvcc.cc:205] Tried to move back new op lower bound from 7023506393214828544 to 7023506392194019328. Current Snapshot: MvccSnapshot[applied={T|T < 7023506393214828544}]
INSERT report
rows total: 2000
time total: 219.967 ms
time per row: 0.109983 ms
I20240503 08:00:40.707158 17122 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47184:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:40.707701 17122 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_5 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:00:40.713011 17245 tablet_service.cc:1467] Processing CreateTablet for tablet 15a5f02e61d24029b0a5757556085733 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.713330 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a5f02e61d24029b0a5757556085733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.713299 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 15a5f02e61d24029b0a5757556085733 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.713563 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a5f02e61d24029b0a5757556085733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.713652 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 15a5f02e61d24029b0a5757556085733 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.713881 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a5f02e61d24029b0a5757556085733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.715785 19036 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:40.715843 19037 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:40.715940 17906 tablet_service.cc:1467] Processing CreateTablet for tablet 15a5f02e61d24029b0a5757556085733 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.716187 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a5f02e61d24029b0a5757556085733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.716578 19036 tablet_bootstrap.cc:654] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.716578 19037 tablet_bootstrap.cc:654] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.717077 19038 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:40.717837 19038 tablet_bootstrap.cc:654] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.718278 19039 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:40.719004 19039 tablet_bootstrap.cc:654] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.719201 17774 tablet_service.cc:1467] Processing CreateTablet for tablet 15a5f02e61d24029b0a5757556085733 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.719447 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a5f02e61d24029b0a5757556085733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.720734 19039 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:40.720811 19039 ts_tablet_manager.cc:1367] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:00:40.721263 19039 raft_consensus.cc:348] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.721292 18038 tablet_service.cc:1467] Processing CreateTablet for tablet 15a5f02e61d24029b0a5757556085733 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.721372 19039 raft_consensus.cc:374] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.721400 19039 raft_consensus.cc:729] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.721490 17905 tablet_service.cc:1467] Processing CreateTablet for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.721552 18038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a5f02e61d24029b0a5757556085733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.721570 19042 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:40.721544 19039 consensus_queue.cc:260] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.721701 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe0edacc8ce4ba8be5e533c82ea92e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.721818 19039 ts_tablet_manager.cc:1398] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.722127 18036 tablet_service.cc:1467] Processing CreateTablet for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.722324 18036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe0edacc8ce4ba8be5e533c82ea92e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.722651 19042 tablet_bootstrap.cc:654] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.723518 18037 tablet_service.cc:1467] Processing CreateTablet for tablet 304ad5f1abf44548a72961d3f655cf61 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.723709 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304ad5f1abf44548a72961d3f655cf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.724107 19040 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:40.724803 19040 tablet_bootstrap.cc:654] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.725143 17642 tablet_service.cc:1467] Processing CreateTablet for tablet 15a5f02e61d24029b0a5757556085733 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.725400 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a5f02e61d24029b0a5757556085733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.725405 17641 tablet_service.cc:1467] Processing CreateTablet for tablet 304ad5f1abf44548a72961d3f655cf61 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.725804 17906 tablet_service.cc:1467] Processing CreateTablet for tablet 304ad5f1abf44548a72961d3f655cf61 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.726011 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304ad5f1abf44548a72961d3f655cf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.726033 17508 tablet_service.cc:1467] Processing CreateTablet for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.726275 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe0edacc8ce4ba8be5e533c82ea92e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.726611 17641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304ad5f1abf44548a72961d3f655cf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.726717 17640 tablet_service.cc:1467] Processing CreateTablet for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.726925 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe0edacc8ce4ba8be5e533c82ea92e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.728245 19040 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:40.728260 17376 tablet_service.cc:1467] Processing CreateTablet for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.728296 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 304ad5f1abf44548a72961d3f655cf61 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.728322 19040 ts_tablet_manager.cc:1367] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:40.728500 17376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe0edacc8ce4ba8be5e533c82ea92e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.728709 19040 raft_consensus.cc:348] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.728821 19040 raft_consensus.cc:374] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.728850 19040 raft_consensus.cc:729] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.728977 19038 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:40.728960 19040 consensus_queue.cc:260] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.729048 19038 ts_tablet_manager.cc:1367] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 08:00:40.729144 19042 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:40.729220 19042 ts_tablet_manager.cc:1367] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:00:40.729249 19040 ts_tablet_manager.cc:1398] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.729403 19040 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:40.729490 19038 raft_consensus.cc:348] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.729604 19038 raft_consensus.cc:374] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.729635 19038 raft_consensus.cc:729] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.729629 19042 raft_consensus.cc:348] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.729735 19042 raft_consensus.cc:374] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.729763 19042 raft_consensus.cc:729] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.729746 19038 consensus_queue.cc:260] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.729871 19042 consensus_queue.cc:260] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.730005 19038 ts_tablet_manager.cc:1398] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.730124 19042 ts_tablet_manager.cc:1398] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.730141 19040 tablet_bootstrap.cc:654] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.731026 17244 tablet_service.cc:1467] Processing CreateTablet for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.731279 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe0edacc8ce4ba8be5e533c82ea92e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.731871 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304ad5f1abf44548a72961d3f655cf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.732121 17774 tablet_service.cc:1467] Processing CreateTablet for tablet 304ad5f1abf44548a72961d3f655cf61 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.732352 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304ad5f1abf44548a72961d3f655cf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.734261 17772 tablet_service.cc:1467] Processing CreateTablet for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.731026 17245 tablet_service.cc:1467] Processing CreateTablet for tablet 304ad5f1abf44548a72961d3f655cf61 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.734484 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe0edacc8ce4ba8be5e533c82ea92e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.734601 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304ad5f1abf44548a72961d3f655cf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.735253 19041 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:40.736106 19041 tablet_bootstrap.cc:654] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.736950 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 304ad5f1abf44548a72961d3f655cf61 (DEFAULT_TABLE table=rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:40.737028 19041 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:40.737139 19041 ts_tablet_manager.cc:1367] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:00:40.737236 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304ad5f1abf44548a72961d3f655cf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:40.737516 19041 raft_consensus.cc:348] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.737625 19041 raft_consensus.cc:374] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.737658 19041 raft_consensus.cc:729] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.737761 19041 consensus_queue.cc:260] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.738027 19041 ts_tablet_manager.cc:1398] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.738052 19042 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:40.738181 19041 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:40.738759 19041 tablet_bootstrap.cc:654] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.738896 19042 tablet_bootstrap.cc:654] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.739404 19038 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:40.739553 19040 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:40.739635 19040 ts_tablet_manager.cc:1367] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 08:00:40.740010 19040 raft_consensus.cc:348] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.740130 19040 raft_consensus.cc:374] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.740161 19040 raft_consensus.cc:729] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.740237 19042 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:40.740306 19042 ts_tablet_manager.cc:1367] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:40.740267 19040 consensus_queue.cc:260] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.740479 19038 tablet_bootstrap.cc:654] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.740547 19040 ts_tablet_manager.cc:1398] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.740701 19040 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:40.741034 19039 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:40.740976 19042 raft_consensus.cc:348] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.741102 19042 raft_consensus.cc:374] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.741132 19042 raft_consensus.cc:729] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.741253 19042 consensus_queue.cc:260] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.741497 19040 tablet_bootstrap.cc:654] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.741539 19042 ts_tablet_manager.cc:1398] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.741937 19039 tablet_bootstrap.cc:654] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.742864 19040 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:40.742949 19040 ts_tablet_manager.cc:1367] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:40.743023 19038 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:40.743135 19038 ts_tablet_manager.cc:1367] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:00:40.743352 19040 raft_consensus.cc:348] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.743480 19040 raft_consensus.cc:374] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.743518 19040 raft_consensus.cc:729] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.743649 19040 consensus_queue.cc:260] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.743721 19038 raft_consensus.cc:348] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.743942 19038 raft_consensus.cc:374] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.743974 19040 ts_tablet_manager.cc:1398] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.744060 19038 raft_consensus.cc:729] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.744150 19042 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:40.744275 19038 consensus_queue.cc:260] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.744985 19042 tablet_bootstrap.cc:654] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.745007 19041 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:40.745121 19041 ts_tablet_manager.cc:1367] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:00:40.745211 19038 ts_tablet_manager.cc:1398] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:00:40.745415 19038 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:40.745548 19041 raft_consensus.cc:348] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.745646 19041 raft_consensus.cc:374] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.745666 19041 raft_consensus.cc:729] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.745776 19041 consensus_queue.cc:260] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.746018 19041 ts_tablet_manager.cc:1398] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:40.746171 19038 tablet_bootstrap.cc:654] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.746706 19041 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:40.747509 19041 tablet_bootstrap.cc:654] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.748438 19038 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:40.748514 19038 ts_tablet_manager.cc:1367] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:00:40.748654 19037 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:40.748747 19037 ts_tablet_manager.cc:1367] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240503 08:00:40.748986 19038 raft_consensus.cc:348] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.749115 19038 raft_consensus.cc:374] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.749151 19038 raft_consensus.cc:729] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.749161 19037 raft_consensus.cc:348] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.749293 19037 raft_consensus.cc:374] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.749336 19037 raft_consensus.cc:729] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.749284 19038 consensus_queue.cc:260] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.749471 19037 consensus_queue.cc:260] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.749605 19038 ts_tablet_manager.cc:1398] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.749763 19037 ts_tablet_manager.cc:1398] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.749925 19037 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:40.750195 19042 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:40.750384 19042 ts_tablet_manager.cc:1367] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:40.750687 19037 tablet_bootstrap.cc:654] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.750792 19041 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:40.750800 19042 raft_consensus.cc:348] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.750874 19041 ts_tablet_manager.cc:1367] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:40.750916 19042 raft_consensus.cc:374] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.750952 19042 raft_consensus.cc:729] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.751070 19042 consensus_queue.cc:260] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.751297 19041 raft_consensus.cc:348] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.751415 19041 raft_consensus.cc:374] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.751452 19041 raft_consensus.cc:729] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.751566 19041 consensus_queue.cc:260] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.751824 19041 ts_tablet_manager.cc:1398] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.752768 19042 ts_tablet_manager.cc:1398] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:40.753170 19039 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:40.753273 19039 ts_tablet_manager.cc:1367] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 08:00:40.753720 19039 raft_consensus.cc:348] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.753832 19039 raft_consensus.cc:374] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.753865 19039 raft_consensus.cc:729] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.753970 19039 consensus_queue.cc:260] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.754204 19039 ts_tablet_manager.cc:1398] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.754344 19039 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:40.755612 19039 tablet_bootstrap.cc:654] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.756451 19039 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:40.756457 19036 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:40.756528 19039 ts_tablet_manager.cc:1367] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:40.756533 19036 ts_tablet_manager.cc:1367] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.001s
I20240503 08:00:40.756839 19039 raft_consensus.cc:348] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.756939 19039 raft_consensus.cc:374] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.756887 19036 raft_consensus.cc:348] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.756969 19039 raft_consensus.cc:729] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.757007 19036 raft_consensus.cc:374] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.757040 19036 raft_consensus.cc:729] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.757107 19039 consensus_queue.cc:260] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.757175 19036 consensus_queue.cc:260] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.757436 19036 ts_tablet_manager.cc:1398] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.757597 19036 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:40.758080 19039 ts_tablet_manager.cc:1398] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:40.758428 19036 tablet_bootstrap.cc:654] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.758677 19037 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:40.758734 19037 ts_tablet_manager.cc:1367] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 08:00:40.759088 19037 raft_consensus.cc:348] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.759191 19037 raft_consensus.cc:374] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.759212 19037 raft_consensus.cc:729] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.759317 19037 consensus_queue.cc:260] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.759593 19037 ts_tablet_manager.cc:1398] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.759716 19037 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:40.760601 19037 tablet_bootstrap.cc:654] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.760850 19036 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:40.760922 19036 ts_tablet_manager.cc:1367] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:00:40.761440 19036 raft_consensus.cc:348] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.761554 19036 raft_consensus.cc:374] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.761580 19036 raft_consensus.cc:729] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.761618 19037 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:40.761698 19037 ts_tablet_manager.cc:1367] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:40.761713 19036 consensus_queue.cc:260] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.761992 19036 ts_tablet_manager.cc:1398] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.762182 19036 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:40.762297 19037 raft_consensus.cc:348] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.762429 19037 raft_consensus.cc:374] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.762465 19037 raft_consensus.cc:729] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.762782 19037 consensus_queue.cc:260] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.763028 19037 ts_tablet_manager.cc:1398] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:40.763581 19036 tablet_bootstrap.cc:654] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:40.764462 19036 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:40.764537 19036 ts_tablet_manager.cc:1367] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:00:40.764911 19036 raft_consensus.cc:348] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.765018 19036 raft_consensus.cc:374] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:40.765046 19036 raft_consensus.cc:729] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:40.765197 19036 consensus_queue.cc:260] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.765471 19036 ts_tablet_manager.cc:1398] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:40.775496 18551 raft_consensus.cc:482] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:40.775596 18551 raft_consensus.cc:504] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.776180 18551 leader_election.cc:288] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:40.776464 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:40.776477 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:40.776504 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:40.776602 18057 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 0.
I20240503 08:00:40.776607 17660 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 0.
I20240503 08:00:40.776607 17398 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 0.
I20240503 08:00:40.776741 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:40.776803 17265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:40.776865 17925 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 0.
I20240503 08:00:40.776913 17265 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 0.
I20240503 08:00:40.777127 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:40.777190 17728 leader_election.cc:302] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:40.777257 17529 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 0.
I20240503 08:00:40.777352 18551 raft_consensus.cc:2781] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:40.777405 18551 raft_consensus.cc:482] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:40.777441 18551 raft_consensus.cc:3037] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.778298 18551 raft_consensus.cc:504] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.778810 18551 leader_election.cc:288] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [CANDIDATE]: Term 1 election: Requested vote from peers 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:40.779047 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:40.779162 18057 raft_consensus.cc:3037] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.779335 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:40.779439 17398 raft_consensus.cc:3037] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.779426 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:40.779518 17925 raft_consensus.cc:3037] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.780154 18057 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 1.
I20240503 08:00:40.780287 17398 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 1.
I20240503 08:00:40.780387 17925 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 1.
I20240503 08:00:40.780704 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:40.780810 17660 raft_consensus.cc:3037] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.780882 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:40.780990 17529 raft_consensus.cc:3037] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.781195 17265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe0edacc8ce4ba8be5e533c82ea92e0" candidate_uuid: "84305d92b082406380b107114e4cd61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:40.781299 17265 raft_consensus.cc:3037] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.781715 17660 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 1.
I20240503 08:00:40.781978 17529 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 1.
I20240503 08:00:40.782209 17265 raft_consensus.cc:2454] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84305d92b082406380b107114e4cd61b in term 1.
I20240503 08:00:40.782408 17727 leader_election.cc:302] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:40.782583 18551 raft_consensus.cc:2781] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:40.782749 18551 raft_consensus.cc:686] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 LEADER]: Becoming Leader. State: Replica: 84305d92b082406380b107114e4cd61b, State: Running, Role: LEADER
I20240503 08:00:40.783059 18551 consensus_queue.cc:237] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:40.785836 17122 catalog_manager.cc:5484] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b reported cstate change: term changed from 0 to 1, leader changed from <none> to 84305d92b082406380b107114e4cd61b (127.0.99.133). New cstate: current_term: 1 leader_uuid: "84305d92b082406380b107114e4cd61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:40.800530 19016 raft_consensus.cc:482] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:40.800648 19016 raft_consensus.cc:504] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.801277 19016 leader_election.cc:288] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:40.801584 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:40.801717 17660 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.801960 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:40.802081 18057 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.802140 17265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:40.802261 17265 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.802301 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:40.802405 17925 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.802482 17331 leader_election.cc:302] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:40.802608 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:40.802690 19016 raft_consensus.cc:2781] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:40.802729 19016 raft_consensus.cc:482] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:40.802726 17529 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.802763 19016 raft_consensus.cc:3037] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.802870 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:40.802984 17793 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:40.803683 19016 raft_consensus.cc:504] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.804205 19016 leader_election.cc:288] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:40.804406 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:40.804504 17660 raft_consensus.cc:3037] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.805371 17660 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.805702 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:40.805757 17265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:40.805802 17793 raft_consensus.cc:3037] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.805877 17265 raft_consensus.cc:3037] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.806553 17265 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.806635 17793 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.806914 17331 leader_election.cc:302] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:40.807060 19016 raft_consensus.cc:2781] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:40.807155 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:40.807207 19016 raft_consensus.cc:686] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Becoming Leader. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Running, Role: LEADER
I20240503 08:00:40.807261 18057 raft_consensus.cc:3037] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.807343 19016 consensus_queue.cc:237] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.808041 18057 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.808405 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:40.808530 17925 raft_consensus.cc:3037] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.809405 17925 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.809660 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304ad5f1abf44548a72961d3f655cf61" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:40.809777 17529 raft_consensus.cc:3037] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.809762 17122 catalog_manager.cc:5484] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:40.810724 17529 raft_consensus.cc:2454] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:40.820070 19106 raft_consensus.cc:482] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:40.820195 19106 raft_consensus.cc:504] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.820773 19106 leader_election.cc:288] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:40.821168 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:40.821185 17265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:40.821184 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:40.821257 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:40.821306 17265 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.821306 17398 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.821309 17660 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.821362 17529 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.821568 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:40.821688 17793 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.821681 17859 leader_election.cc:302] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, 8eb6822290b046f88c861159aa9421ec; no voters:
I20240503 08:00:40.821688 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:40.821805 18057 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:40.821898 19106 raft_consensus.cc:2781] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:40.821949 19106 raft_consensus.cc:482] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:40.821975 19106 raft_consensus.cc:3037] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.822718 19106 raft_consensus.cc:504] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.823159 19106 leader_election.cc:288] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:40.823408 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:40.823513 17793 raft_consensus.cc:3037] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.823509 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:40.823609 18057 raft_consensus.cc:3037] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.823674 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:40.823767 17660 raft_consensus.cc:3037] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.823783 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:40.823879 17398 raft_consensus.cc:3037] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.824376 17793 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.824535 18057 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.824566 17660 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.824738 17398 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.824801 17265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:40.824838 17859 leader_election.cc:302] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:40.824875 17529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15a5f02e61d24029b0a5757556085733" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:40.824913 17265 raft_consensus.cc:3037] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.824971 17529 raft_consensus.cc:3037] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:40.825032 19106 raft_consensus.cc:2781] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:40.825129 19106 raft_consensus.cc:686] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Becoming Leader. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Running, Role: LEADER
I20240503 08:00:40.825275 19106 consensus_queue.cc:237] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:40.825778 17265 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.825829 17529 raft_consensus.cc:2454] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:40.827266 17122 catalog_manager.cc:5484] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:40.935142 19160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:40.935398 19160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:40.938566 19160 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 08:00:41.011147 18057 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.011233 17529 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.011402 17660 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.011655 17265 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.011814 19017 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.012019 18927 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.012282 18927 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.013259 17662 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.013307 17793 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.013687 18815 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.013967 18815 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.016106 17794 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.016881 17265 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.017342 17528 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.017735 17398 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.017915 19107 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.018378 19107 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.018838 18927 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.018954 17925 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.020273 18927 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.029893 18927 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.032105 18057 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.032572 19106 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.033802 17528 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.037667 17924 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.037969 17398 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.039232 18551 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.041266 18057 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.041518 19107 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.044824 17265 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.045924 19156 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.046259 19156 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.047125 17662 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.047938 19156 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.048234 19156 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.059451 18551 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 163.897 ms
time per row: 0.0819487 ms
I20240503 08:00:41.155530 17116 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47184:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:00:41.156050 17116 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_6 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240503 08:00:41.167382 17906 tablet_service.cc:1467] Processing CreateTablet for tablet 8500b198108649a7bcb614c93cf53a3a (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.167646 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.169363 17508 tablet_service.cc:1467] Processing CreateTablet for tablet e57bf40798824baf8050988874664154 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.169410 18037 tablet_service.cc:1467] Processing CreateTablet for tablet 8500b198108649a7bcb614c93cf53a3a (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.169652 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.169667 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.169904 17509 tablet_service.cc:1467] Processing CreateTablet for tablet 882c64fe082243fbbb3099f2bf6e33e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.170084 17509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882c64fe082243fbbb3099f2bf6e33e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.171628 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 8500b198108649a7bcb614c93cf53a3a (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.171803 19036 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:41.171824 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.172595 19036 tablet_bootstrap.cc:654] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.173561 19036 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:41.173581 19040 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:41.173641 19036 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:41.174063 19036 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.174196 19036 raft_consensus.cc:374] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.174228 19036 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.174332 19036 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.174607 19036 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:41.174645 19040 tablet_bootstrap.cc:654] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.174757 19036 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:41.175046 17378 tablet_service.cc:1467] Processing CreateTablet for tablet 8500b198108649a7bcb614c93cf53a3a (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.175292 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.175529 19036 tablet_bootstrap.cc:654] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.175858 17245 tablet_service.cc:1467] Processing CreateTablet for tablet 8500b198108649a7bcb614c93cf53a3a (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.176077 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.177419 17376 tablet_service.cc:1467] Processing CreateTablet for tablet e57bf40798824baf8050988874664154 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.177650 17376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.177846 17772 tablet_service.cc:1467] Processing CreateTablet for tablet 8500b198108649a7bcb614c93cf53a3a (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.178067 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.178043 17244 tablet_service.cc:1467] Processing CreateTablet for tablet e57bf40798824baf8050988874664154 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.178268 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.179996 17246 tablet_service.cc:1467] Processing CreateTablet for tablet 882c64fe082243fbbb3099f2bf6e33e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.180058 17774 tablet_service.cc:1467] Processing CreateTablet for tablet e57bf40798824baf8050988874664154 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.180187 17246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882c64fe082243fbbb3099f2bf6e33e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.180275 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.181203 19038 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:41.181281 17773 tablet_service.cc:1467] Processing CreateTablet for tablet 882c64fe082243fbbb3099f2bf6e33e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.181501 17773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882c64fe082243fbbb3099f2bf6e33e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.177419 17377 tablet_service.cc:1467] Processing CreateTablet for tablet 882c64fe082243fbbb3099f2bf6e33e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.181710 17377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882c64fe082243fbbb3099f2bf6e33e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.181916 19038 tablet_bootstrap.cc:654] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.182443 19037 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:41.183215 19037 tablet_bootstrap.cc:654] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.183871 19042 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:41.184630 19042 tablet_bootstrap.cc:654] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.185165 19037 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:41.185256 19037 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:00:41.185657 19037 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.185766 19037 raft_consensus.cc:374] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.185801 19037 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.185745 18036 tablet_service.cc:1467] Processing CreateTablet for tablet 882c64fe082243fbbb3099f2bf6e33e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.185910 19037 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.185969 18036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882c64fe082243fbbb3099f2bf6e33e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.186203 19037 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.186354 19037 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:41.187116 19037 tablet_bootstrap.cc:654] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.187515 17904 tablet_service.cc:1467] Processing CreateTablet for tablet 882c64fe082243fbbb3099f2bf6e33e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.187752 17904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882c64fe082243fbbb3099f2bf6e33e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.187760 17641 tablet_service.cc:1467] Processing CreateTablet for tablet 8500b198108649a7bcb614c93cf53a3a (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.187987 17641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.188436 18037 tablet_service.cc:1467] Processing CreateTablet for tablet e57bf40798824baf8050988874664154 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.188654 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.187515 17905 tablet_service.cc:1467] Processing CreateTablet for tablet e57bf40798824baf8050988874664154 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.189915 19041 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:41.190044 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.190061 17642 tablet_service.cc:1467] Processing CreateTablet for tablet e57bf40798824baf8050988874664154 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.190248 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.190676 19041 tablet_bootstrap.cc:654] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.192071 17640 tablet_service.cc:1467] Processing CreateTablet for tablet 882c64fe082243fbbb3099f2bf6e33e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:00:41.192265 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882c64fe082243fbbb3099f2bf6e33e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:41.192703 19042 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:41.192777 19042 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 08:00:41.192844 19037 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:41.192911 19037 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:00:41.193125 19040 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:41.193202 19040 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.000s
I20240503 08:00:41.193230 19042 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.193332 19042 raft_consensus.cc:374] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.193360 19042 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.193325 19037 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.193420 19037 raft_consensus.cc:374] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.193446 19037 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.193470 19042 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.193547 19037 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.193606 19040 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.193698 19040 raft_consensus.cc:374] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.193728 19040 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.193733 19042 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:41.193802 19037 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.193881 19042 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:41.193837 19040 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.193936 19037 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec: Bootstrap starting.
I20240503 08:00:41.194036 19039 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:41.194095 19040 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:41.194272 19040 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:41.194705 19037 tablet_bootstrap.cc:654] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.194795 19039 tablet_bootstrap.cc:654] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.194851 19042 tablet_bootstrap.cc:654] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.195075 19040 tablet_bootstrap.cc:654] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.195978 19038 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:41.196056 19038 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 08:00:41.196481 19038 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.196633 19038 raft_consensus.cc:374] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.196671 19038 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.196770 19038 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.197024 19038 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.197480 19041 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:41.197551 19041 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 08:00:41.198026 19038 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:41.197964 19041 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.198073 19041 raft_consensus.cc:374] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.198103 19041 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.198230 19041 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.198508 19041 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.198661 19041 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:41.199440 19038 tablet_bootstrap.cc:654] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.199443 19041 tablet_bootstrap.cc:654] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.200350 19036 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:41.200448 19036 ts_tablet_manager.cc:1367] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
I20240503 08:00:41.200855 19036 raft_consensus.cc:348] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.200987 19036 raft_consensus.cc:374] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.201022 19036 raft_consensus.cc:729] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.201155 19036 consensus_queue.cc:260] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.201424 19036 ts_tablet_manager.cc:1398] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.201483 19038 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:41.201546 19038 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:41.201612 19036 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:41.201901 19038 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.202016 19038 raft_consensus.cc:374] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.202052 19038 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.202152 19038 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.202323 19036 tablet_bootstrap.cc:654] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.202409 19038 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.202564 19038 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:41.203348 19038 tablet_bootstrap.cc:654] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.203727 19039 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:41.203812 19039 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 08:00:41.204164 19040 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:41.204236 19040 ts_tablet_manager.cc:1367] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 08:00:41.204176 19039 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.204288 19039 raft_consensus.cc:374] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.204304 19037 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec: No bootstrap required, opened a new log
I20240503 08:00:41.204320 19039 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.204368 19037 ts_tablet_manager.cc:1367] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 08:00:41.204430 19039 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.204586 19040 raft_consensus.cc:348] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.204690 19040 raft_consensus.cc:374] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.204699 19039 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.204722 19040 raft_consensus.cc:729] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.204716 19037 raft_consensus.cc:348] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.204820 19037 raft_consensus.cc:374] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.204852 19039 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:41.204849 19037 raft_consensus.cc:729] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6822290b046f88c861159aa9421ec, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.204829 19040 consensus_queue.cc:260] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.204950 19037 consensus_queue.cc:260] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.205152 19040 ts_tablet_manager.cc:1398] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 08:00:41.205233 19037 ts_tablet_manager.cc:1398] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:00:41.205304 19040 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d: Bootstrap starting.
I20240503 08:00:41.205642 19039 tablet_bootstrap.cc:654] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.206075 19038 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298: No bootstrap required, opened a new log
I20240503 08:00:41.206116 19040 tablet_bootstrap.cc:654] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.206149 19038 ts_tablet_manager.cc:1367] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:00:41.206530 19038 raft_consensus.cc:348] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.206630 19038 raft_consensus.cc:374] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.206661 19038 raft_consensus.cc:729] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.206758 19038 consensus_queue.cc:260] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.207010 19038 ts_tablet_manager.cc:1398] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.207643 19036 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc: No bootstrap required, opened a new log
I20240503 08:00:41.207660 19040 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d: No bootstrap required, opened a new log
I20240503 08:00:41.207720 19036 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:41.207720 19040 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:00:41.208082 19036 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.208201 19036 raft_consensus.cc:374] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.208231 19036 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.208181 19040 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.208278 19040 raft_consensus.cc:374] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.208310 19040 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d502c5c7f12a4e39aa25cd57f92c403d, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.208338 19036 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.208392 19040 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.208606 19036 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:41.208622 19041 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:41.208694 19041 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 08:00:41.209112 19041 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.209228 19041 raft_consensus.cc:374] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.209259 19041 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.209408 19041 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.209666 19041 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.209830 19041 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:41.210558 19039 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:41.210637 19039 ts_tablet_manager.cc:1367] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 08:00:41.210664 19041 tablet_bootstrap.cc:654] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.211119 19039 raft_consensus.cc:348] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.211261 19039 raft_consensus.cc:374] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.211300 19039 raft_consensus.cc:729] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.211402 19039 consensus_queue.cc:260] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.211692 19039 ts_tablet_manager.cc:1398] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.211840 19039 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: Bootstrap starting.
I20240503 08:00:41.212005 19041 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5: No bootstrap required, opened a new log
I20240503 08:00:41.212078 19041 ts_tablet_manager.cc:1367] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:41.212639 19039 tablet_bootstrap.cc:654] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.212639 19041 raft_consensus.cc:348] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.212734 19041 raft_consensus.cc:374] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.212764 19041 raft_consensus.cc:729] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.212872 19041 consensus_queue.cc:260] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.213510 19041 ts_tablet_manager.cc:1398] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.214610 19042 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:41.214699 19042 ts_tablet_manager.cc:1367] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 08:00:41.215250 19040 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:00:41.215490 19042 raft_consensus.cc:348] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.215615 19042 raft_consensus.cc:374] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.215651 19042 raft_consensus.cc:729] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.215795 19042 consensus_queue.cc:260] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.215988 19039 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: No bootstrap required, opened a new log
I20240503 08:00:41.216063 19039 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:00:41.216240 19042 ts_tablet_manager.cc:1398] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.216476 19039 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.216610 19039 raft_consensus.cc:374] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.216656 19039 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.216761 19039 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.217049 19039 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.217185 19042 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:41.218616 19042 tablet_bootstrap.cc:654] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b: Neither blocks nor log segments found. Creating new log.
I20240503 08:00:41.219780 19042 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b: No bootstrap required, opened a new log
I20240503 08:00:41.219866 19042 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:00:41.220173 19042 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.220268 19042 raft_consensus.cc:374] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:00:41.220297 19042 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: FOLLOWER
I20240503 08:00:41.220525 19042 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.220805 19042 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:41.236069 19188 raft_consensus.cc:482] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:41.236167 19188 raft_consensus.cc:504] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.236828 19188 leader_election.cc:288] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:41.236974 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:41.237128 17924 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:41.237156 18056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:41.237304 18056 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:41.237533 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:41.237655 17528 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:41.237630 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:41.237748 17266 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:41.237915 17331 leader_election.cc:302] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:41.238129 19188 raft_consensus.cc:2781] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:41.238111 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:41.238135 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:41.238190 19188 raft_consensus.cc:482] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:41.238229 19188 raft_consensus.cc:3037] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.238229 17794 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:41.238236 17660 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 0.
I20240503 08:00:41.239140 19188 raft_consensus.cc:504] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.239666 19188 leader_election.cc:288] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Requested vote from peers 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:41.239810 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:41.239902 17924 raft_consensus.cc:3037] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.239950 18056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:41.240056 18056 raft_consensus.cc:3037] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.240351 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:41.240458 17660 raft_consensus.cc:3037] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.240603 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:41.240697 17266 raft_consensus.cc:3037] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.240712 17924 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:41.240994 18056 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:41.241266 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:41.241348 17660 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:41.241372 17528 raft_consensus.cc:3037] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.241609 17331 leader_election.cc:302] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, d502c5c7f12a4e39aa25cd57f92c403d, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:41.241686 17266 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:41.241747 19188 raft_consensus.cc:2781] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:41.241815 19188 raft_consensus.cc:686] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Becoming Leader. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Running, Role: LEADER
I20240503 08:00:41.241838 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500b198108649a7bcb614c93cf53a3a" candidate_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:41.241920 17794 raft_consensus.cc:3037] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.241940 19188 consensus_queue.cc:237] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:41.242339 17528 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:41.242862 17794 raft_consensus.cc:2454] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a940e3a6e340a69ec7ac31d26c9298 in term 1.
I20240503 08:00:41.244048 17118 catalog_manager.cc:5484] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:41.260831 19106 raft_consensus.cc:482] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:41.260931 19106 raft_consensus.cc:504] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.261533 19106 leader_election.cc:288] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:41.261655 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:41.261778 17266 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:41.261873 18056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:41.261993 18056 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:41.262166 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:41.262248 17794 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:41.262508 17861 leader_election.cc:302] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:41.262542 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:41.262631 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5" is_pre_election: true
I20240503 08:00:41.262661 17397 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:41.262676 19106 raft_consensus.cc:2781] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:41.262717 19106 raft_consensus.cc:482] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:41.262754 19106 raft_consensus.cc:3037] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.262749 17660 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:41.263025 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:41.263136 17528 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 0.
I20240503 08:00:41.263672 19106 raft_consensus.cc:504] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.264132 19106 leader_election.cc:288] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:41.264247 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:41.264341 17266 raft_consensus.cc:3037] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.264348 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:41.264456 17794 raft_consensus.cc:3037] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.264595 18056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:41.264613 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:41.264684 18056 raft_consensus.cc:3037] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.264688 17397 raft_consensus.cc:3037] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.265252 17794 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:41.265252 17266 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:41.265544 18056 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:41.265544 17397 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:41.265794 17859 leader_election.cc:302] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 78e52cf3c8fa435bb5ca36061227ac41, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, f0a940e3a6e340a69ec7ac31d26c9298; no voters:
I20240503 08:00:41.265853 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce150b975ff4040bedde6319d5cf3d5"
I20240503 08:00:41.265949 19106 raft_consensus.cc:2781] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:41.265957 17660 raft_consensus.cc:3037] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.266024 19106 raft_consensus.cc:686] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Becoming Leader. State: Replica: 78e52cf3c8fa435bb5ca36061227ac41, State: Running, Role: LEADER
I20240503 08:00:41.266155 19106 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.266752 17660 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:41.266952 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57bf40798824baf8050988874664154" candidate_uuid: "78e52cf3c8fa435bb5ca36061227ac41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:41.267066 17528 raft_consensus.cc:3037] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.267882 17528 raft_consensus.cc:2454] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e52cf3c8fa435bb5ca36061227ac41 in term 1.
I20240503 08:00:41.268054 17118 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:41.271507 19235 raft_consensus.cc:482] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:00:41.271592 19235 raft_consensus.cc:504] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.272279 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec" is_pre_election: true
I20240503 08:00:41.272341 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc" is_pre_election: true
I20240503 08:00:41.272390 17266 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:41.272392 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41" is_pre_election: true
I20240503 08:00:41.272459 17528 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:41.272478 17924 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:41.272606 17397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" is_pre_election: true
I20240503 08:00:41.272712 17397 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:41.272842 17596 leader_election.cc:302] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 78e52cf3c8fa435bb5ca36061227ac41, 7af8f2a481294a2cb93e5761947978cc, 8eb6822290b046f88c861159aa9421ec; no voters:
I20240503 08:00:41.273018 19010 raft_consensus.cc:2781] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:00:41.273095 19010 raft_consensus.cc:482] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:00:41.273141 19010 raft_consensus.cc:3037] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.273171 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b" is_pre_election: true
I20240503 08:00:41.273272 17794 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:41.273550 19235 leader_election.cc:288] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:41.273730 18056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" is_pre_election: true
I20240503 08:00:41.273828 18056 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 0.
I20240503 08:00:41.274060 19010 raft_consensus.cc:504] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.274605 19010 leader_election.cc:288] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 election: Requested vote from peers f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225), 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199), 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647), 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175), 84305d92b082406380b107114e4cd61b (127.0.99.133:38825), d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:41.274895 17266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6822290b046f88c861159aa9421ec"
I20240503 08:00:41.274928 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84305d92b082406380b107114e4cd61b"
I20240503 08:00:41.274989 17266 raft_consensus.cc:3037] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.274991 18056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d502c5c7f12a4e39aa25cd57f92c403d"
I20240503 08:00:41.275013 17794 raft_consensus.cc:3037] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.274986 17398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a940e3a6e340a69ec7ac31d26c9298"
I20240503 08:00:41.275069 18056 raft_consensus.cc:3037] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.275072 17398 raft_consensus.cc:3037] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.275835 17794 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:41.275835 17398 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:41.275847 18056 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:41.275847 17266 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:41.276177 17595 leader_election.cc:302] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0ce150b975ff4040bedde6319d5cf3d5, 84305d92b082406380b107114e4cd61b, 8eb6822290b046f88c861159aa9421ec, d502c5c7f12a4e39aa25cd57f92c403d; no voters:
I20240503 08:00:41.276291 17924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e52cf3c8fa435bb5ca36061227ac41"
I20240503 08:00:41.276336 19010 raft_consensus.cc:2781] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:00:41.276394 17924 raft_consensus.cc:3037] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.276417 19010 raft_consensus.cc:686] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Running, Role: LEADER
I20240503 08:00:41.276566 17528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882c64fe082243fbbb3099f2bf6e33e6" candidate_uuid: "0ce150b975ff4040bedde6319d5cf3d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8f2a481294a2cb93e5761947978cc"
I20240503 08:00:41.276552 19010 consensus_queue.cc:237] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:41.276661 17528 raft_consensus.cc:3037] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:00:41.277263 17924 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:41.277508 17528 raft_consensus.cc:2454] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ce150b975ff4040bedde6319d5cf3d5 in term 1.
I20240503 08:00:41.278676 17118 catalog_manager.cc:5484] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
W20240503 08:00:41.400768 19240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:41.401001 19240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:41.404278 19240 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 08:00:41.483534 17266 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.483762 17528 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.484192 19188 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.484463 19188 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.492123 17660 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.492643 19106 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.492731 17530 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.492766 17265 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.493188 17397 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.493257 18056 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.493264 17794 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.493817 19065 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.494105 19065 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.494287 19106 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.494489 17924 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.494828 17794 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.494972 19188 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.495215 18056 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.495260 19188 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.494491 19065 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240503 08:00:41.502169 18815 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.505003 19017 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.506139 17662 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.508256 19017 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.518431 17791 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.518617 17925 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.518803 19010 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.519088 19010 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.520182 17529 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.520491 17398 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.520998 19235 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.521968 18058 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.522356 19235 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.524835 19285 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:00:41.528425 17264 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:00:41.530722 19235 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 186.275 ms
time per row: 0.0931377 ms
I20240503 08:00:41.640377 18991 mvcc.cc:205] Tried to move back new op lower bound from 7023506397255131136 to 7023506396271341568. Current Snapshot: MvccSnapshot[applied={T|T < 7023506397255131136}]
I20240503 08:00:41.651702 18985 mvcc.cc:205] Tried to move back new op lower bound from 7023506397255131136 to 7023506396271341568. Current Snapshot: MvccSnapshot[applied={T|T < 7023506397255131136}]
I20240503 08:00:41.665447 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.137:44583
--local_ip_for_outbound_sockets=127.0.99.137
--tserver_master_addrs=127.0.99.190:35233
--webserver_port=34845
--webserver_interface=127.0.99.137
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:00:41.775388 19292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:41.775610 19292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:41.775669 19292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:41.778865 19292 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:41.778924 19292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:41.779003 19292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.137
I20240503 08:00:41.783071 19292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.137:44583
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.0.99.137
--webserver_port=34845
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.19292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.137
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:41.783906 19292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:41.784955 19292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:41.793741 19297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:41.794531 19298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:41.795205 19300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:41.795967 19292 server_base.cc:1034] running on GCE node
I20240503 08:00:41.796316 19292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:41.796834 19292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:41.798022 19292 hybrid_clock.cc:648] HybridClock initialized: now 1714723241797986 us; error 45 us; skew 500 ppm
I20240503 08:00:41.799885 19292 webserver.cc:460] Webserver started at http://127.0.99.137:34845/ using document root <none> and password file <none>
I20240503 08:00:41.800431 19292 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:41.800494 19292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:41.803877 19292 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:41.805723 19306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:41.806715 19292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:41.806809 19292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "0665f4f85b984a8bba213ad479a88bad"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:41.807174 19292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:41.828543 19292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:41.829172 19292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:41.829329 19292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:41.829845 19292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:41.830679 19292 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:41.830724 19292 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:41.830775 19292 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:41.830799 19292 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:41.839071 19292 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.137:44583
I20240503 08:00:41.839121 19419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.137:44583 every 8 connection(s)
I20240503 08:00:41.840001 19292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240503 08:00:41.846081 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 19292
I20240503 08:00:41.846539 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.138:33637
--local_ip_for_outbound_sockets=127.0.99.138
--tserver_master_addrs=127.0.99.190:35233
--webserver_port=43159
--webserver_interface=127.0.99.138
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:41.848865 19420 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:41.849074 19420 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:41.849571 19420 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:41.850648 17115 ts_manager.cc:194] Re-registered known tserver with Master: 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583)
I20240503 08:00:41.851148 17115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.137:60115
W20240503 08:00:41.952010 19423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:41.952208 19423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:41.952270 19423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:41.955377 19423 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:41.955443 19423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:41.955526 19423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.138
I20240503 08:00:41.959051 19423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.138:33637
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.0.99.138
--webserver_port=43159
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.19423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.138
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:41.959874 19423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:41.960917 19423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:41.969550 19429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:41.970330 19428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:41.970985 19431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:41.971665 19423 server_base.cc:1034] running on GCE node
I20240503 08:00:41.972072 19423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:41.972640 19423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:41.973785 19423 hybrid_clock.cc:648] HybridClock initialized: now 1714723241973773 us; error 43 us; skew 500 ppm
I20240503 08:00:41.975772 19423 webserver.cc:460] Webserver started at http://127.0.99.138:43159/ using document root <none> and password file <none>
I20240503 08:00:41.976264 19423 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:41.976333 19423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:41.979799 19423 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 08:00:41.981601 19437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:41.982718 19423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:41.982813 19423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "ab18888a9724446b9ee4fb9efe811b09"
format_stamp: "Formatted at 2024-05-03 08:00:34 on dist-test-slave-7b05"
I20240503 08:00:41.983186 19423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:41.998754 19423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:41.999323 19423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:41.999472 19423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:41.999990 19423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:42.000872 19423 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:42.000926 19423 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:42.000972 19423 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:42.000986 19423 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:42.009001 19423 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.138:33637
I20240503 08:00:42.009037 19550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.138:33637 every 8 connection(s)
I20240503 08:00:42.010037 19423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240503 08:00:42.011730 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 19423
I20240503 08:00:42.012168 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
/tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:41085
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:35233
--webserver_port=37533
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:00:42.019232 19551 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:42.019479 19551 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:42.020099 19551 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:42.021122 17115 ts_manager.cc:194] Re-registered known tserver with Master: ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138:33637)
I20240503 08:00:42.021664 17115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.138:58861
W20240503 08:00:42.125758 19554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:00:42.125996 19554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:00:42.126065 19554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:00:42.129289 19554 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:00:42.129354 19554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:00:42.129434 19554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 08:00:42.133756 19554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:41085
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=37533
--tserver_master_addrs=127.0.99.190:35233
--never_fsync=true
--heap_profile_path=/tmp/kudu.19554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 08:00:42.134685 19554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:00:42.136027 19554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:00:42.145099 19562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:42.145645 19559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:00:42.146692 19560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:00:42.147454 19554 server_base.cc:1034] running on GCE node
I20240503 08:00:42.147809 19554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:00:42.148298 19554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:00:42.149434 19554 hybrid_clock.cc:648] HybridClock initialized: now 1714723242149414 us; error 32 us; skew 500 ppm
I20240503 08:00:42.151264 19554 webserver.cc:460] Webserver started at http://127.0.99.136:37533/ using document root <none> and password file <none>
I20240503 08:00:42.151875 19554 fs_manager.cc:359] Metadata directory not provided
I20240503 08:00:42.151950 19554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:00:42.155689 19554 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:42.157472 19568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:42.158370 19554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 08:00:42.158460 19554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b65378a282b9443484843de0b8906df9"
format_stamp: "Formatted at 2024-05-03 08:00:33 on dist-test-slave-7b05"
I20240503 08:00:42.158821 19554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:00:42.179186 19554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:00:42.179833 19554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:00:42.179980 19554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:00:42.180521 19554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:00:42.181393 19554 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:00:42.181442 19554 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:42.181490 19554 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:00:42.181511 19554 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:00:42.189838 19554 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41085
I20240503 08:00:42.189858 19681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41085 every 8 connection(s)
I20240503 08:00:42.190760 19554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:00:42.198021 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskD2FJ4a/build/debug/bin/kudu as pid 19554
I20240503 08:00:42.199034 19682 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35233
I20240503 08:00:42.199249 19682 heartbeater.cc:464] Registering TS with master...
I20240503 08:00:42.199802 19682 heartbeater.cc:511] Master 127.0.99.190:35233 requested a full tablet report, sending...
I20240503 08:00:42.200786 17115 ts_manager.cc:194] Re-registered known tserver with Master: b65378a282b9443484843de0b8906df9 (127.0.99.136:41085)
I20240503 08:00:42.201484 17115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:53171
I20240503 08:00:42.386927 19354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:42.388440 17640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.388939 17905 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.405573 17774 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.409341 19610 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:42.412601 17510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.412961 18037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.429662 17246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.429817 19485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:42.441676 17377 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:42.546123 17640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.547065 17905 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.549579 17774 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.551180 19354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:42.554504 17246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.564884 19485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:42.584074 19610 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:42.585240 17377 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:42.585366 18037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.593581 17510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.746268 19354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:42.750103 19485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:42.761885 19610 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:42.762594 17774 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.769850 17246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.770205 17905 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.770943 18037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:42.777874 17377 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:42.787477 17510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:42.789692 17640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:42.843567 17265 consensus_queue.cc:237] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:42.845393 17793 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.845600 17398 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.845876 17660 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.845993 19833 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.846240 18058 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.846722 17925 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.846840 17529 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.846900 19833 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.847275 19833 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.847754 19133 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.848037 19133 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.848294 19133 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.850108 19834 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.852113 17793 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.852939 19420 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:42.853560 17119 catalog_manager.cc:5484] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:42.854123 18058 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.854640 17398 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.855558 17660 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.855917 17529 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.859371 17925 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.863780 17529 consensus_queue.cc:237] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:42.865779 18058 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.865783 17398 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.865986 17660 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.866294 19854 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.866531 19855 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.866652 17924 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.867168 17265 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.868373 17793 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.868570 17265 consensus_queue.cc:237] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:42.866874 19854 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.868899 18980 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.869076 19737 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.869246 19856 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.870353 18057 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.870797 17924 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.870973 17397 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.871033 19843 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.871194 19834 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.871253 17662 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.871848 19843 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.872139 19843 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.872385 17791 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.872745 19843 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.874454 19843 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:42.875213 18058 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:42.875393 17791 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:42.876431 17924 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:42.877686 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 99d8d89b8561420781b7e7a9b9c778a0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:42.878149 17398 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:42.878784 17121 catalog_manager.cc:5484] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b reported cstate change: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:42.878885 17662 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
W20240503 08:00:42.879889 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99d8d89b8561420781b7e7a9b9c778a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:42.882401 19737 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:42.883158 17529 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.883037 17925 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:42.883498 19834 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.884575 17529 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:42.885129 17398 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:42.885855 18058 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:42.886447 17118 catalog_manager.cc:5484] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:42.886744 17265 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:42.891732 17529 consensus_queue.cc:237] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:42.891909 17793 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:42.893699 18058 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.894129 17925 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.894188 17660 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.894359 17398 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.894513 19854 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.894589 17265 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.894779 19854 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.895004 19854 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.895936 19854 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.896193 19854 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.896167 17660 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:42.898407 17793 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.898973 19854 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.899421 17662 consensus_queue.cc:237] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:42.900904 19854 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.901585 17398 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.902407 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:42.902815 17925 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.903019 17265 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.903093 17529 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.903164 18058 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.903359 17397 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.903371 17793 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.904223 19284 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.904256 18058 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.904553 19284 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.904690 17265 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.904855 19284 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.905146 19284 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.906049 17925 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.907218 17118 catalog_manager.cc:5484] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.908583 19686 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.908886 19686 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.909288 17660 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.909288 17791 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
W20240503 08:00:42.910662 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a79ddd7d381c4ccbbd1a1e7d3ec8366f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:42.913616 19877 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.914175 17529 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.915186 17793 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.915652 17925 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.917953 17265 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.918121 17121 catalog_manager.cc:5484] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.918421 18058 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.919387 17397 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.925269 17660 consensus_queue.cc:237] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:42.927186 18058 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.927266 17265 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.927549 19877 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.927660 17397 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.928382 19875 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.929116 17925 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.929409 19284 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.929600 17793 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.929931 17529 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.930104 19686 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.930285 19876 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.930408 19686 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:42.931691 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 882c64fe082243fbbb3099f2bf6e33e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:42.932365 19284 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.933140 17529 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.933828 17925 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.933828 17265 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.934685 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 882c64fe082243fbbb3099f2bf6e33e6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:42.936252 17115 catalog_manager.cc:5484] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.936414 17397 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.936424 17793 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.938139 17398 consensus_queue.cc:237] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:42.939996 18058 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.940956 18057 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.941035 17793 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.941099 17265 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.941512 19188 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.941555 17529 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.941855 19188 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.942153 19188 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.942436 19188 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.944031 17660 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.944427 19188 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.944638 17925 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.944953 19017 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.946174 19893 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.946521 18057 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.946805 17529 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.949133 17264 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.949225 17115 catalog_manager.cc:5484] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.949607 17660 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.952539 17925 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.959784 17529 consensus_queue.cc:237] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:42.960008 17398 consensus_queue.cc:237] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:42.959964 17793 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.961987 17660 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.962471 19017 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.962523 17660 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.962636 17925 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.962847 19737 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.963138 19892 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.963368 17925 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.963730 19856 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.963915 17265 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.964365 18058 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.964751 17791 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.964897 19892 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.965762 19737 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.965891 19856 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.966214 17793 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.966598 19894 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.967018 18057 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.967361 19737 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.968052 17529 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.968406 17264 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.968386 19892 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.969013 19188 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.969282 17398 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:42.969667 19737 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.969739 19892 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.971099 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ed32a1099e044448aaa0c9c9f58f74e6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:42.971933 17124 catalog_manager.cc:5484] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:42.972635 18058 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
W20240503 08:00:42.973299 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed32a1099e044448aaa0c9c9f58f74e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:42.972869 17925 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.974937 17530 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.977008 17662 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.978610 19856 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.978786 17791 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.980531 17925 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.981448 17791 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.981858 17120 catalog_manager.cc:5484] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:42.982589 17265 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.982703 17662 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.985580 17530 consensus_queue.cc:237] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:42.986166 17264 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
W20240503 08:00:42.987713 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 143109561ad549d69a59992c02fe23af. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:42.988082 18057 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.988279 17265 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.988312 17925 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.988574 17791 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.988586 18980 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.988677 19737 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.988849 19855 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.989135 19854 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.989512 17397 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.990290 18980 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:42.990271 18057 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.990964 17397 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:42.992486 17662 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:42.992949 18980 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:42.994480 17397 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.993906 18980 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.996383 17124 catalog_manager.cc:5484] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.997196 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 143109561ad549d69a59992c02fe23af with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:42.997233 17925 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.997853 17265 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:42.998615 17791 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.002765 18057 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.005359 17660 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.008468 17264 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:43.010324 17791 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.010897 19835 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.010983 18057 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.011041 17925 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.011409 19835 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.011566 19833 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.013686 17530 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.013701 17660 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.014029 17397 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.014150 19835 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.014499 19834 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.015331 17660 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.016185 17925 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.016775 19835 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.017099 19835 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.017657 17124 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.017817 17397 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.018247 17530 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.020704 17791 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.022560 18057 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.023092 17264 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:43.023929 19551 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:43.025296 17397 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.025599 17925 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.025853 17660 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.026225 18058 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.026244 19133 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.026494 19133 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.026677 17791 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.026665 19836 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.026968 19833 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.027727 19836 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.028142 17530 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.028555 19843 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:43.030637 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138:33637): Couldn't send request to peer ab18888a9724446b9ee4fb9efe811b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a03fdbe7f824674a351fd4598dc2c63. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:43.030783 19834 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.031203 17397 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.032711 17925 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.032732 17791 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.033573 17121 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.033995 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8a03fdbe7f824674a351fd4598dc2c63 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:43.036621 18057 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.038157 17530 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.042047 17660 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.046319 17925 consensus_queue.cc:237] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:43.048087 17791 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.048130 18057 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.048342 17397 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.048547 19936 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.048655 17530 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.048887 19936 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.049198 18815 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.049489 18815 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.050812 17660 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.050812 17264 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.051151 19936 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.051409 19935 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.052165 18815 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.052513 17530 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.054154 18058 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.054512 17791 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.055323 17397 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.056597 17120 catalog_manager.cc:5484] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.058351 17264 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.060806 17660 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.061841 17925 consensus_queue.cc:237] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.064138 17791 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.064348 18058 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.064572 19948 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.065141 17660 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.065436 17397 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.065505 18815 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.065807 17530 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.066200 19934 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.066390 17264 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.066469 19784 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.066751 19784 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.067433 18815 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.069324 18815 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.069882 18058 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.071161 17791 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
W20240503 08:00:43.071617 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70f073cc41d54638a2c51746b94a5ece. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:43.071669 17660 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.072162 17109 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 70f073cc41d54638a2c51746b94a5ece with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:43.072264 17117 catalog_manager.cc:5484] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.073127 17264 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.076687 17397 consensus_queue.cc:237] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:43.077572 17530 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.078646 17529 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.078837 17791 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.079032 19188 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.079326 18058 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.079438 17925 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.079595 17660 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.079916 19188 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.080219 19188 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.080367 19893 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.080474 17397 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:43.080956 17264 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.081360 19893 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.084192 19893 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.085435 19188 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.087002 17662 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.087119 17530 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.088172 17925 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.088351 18058 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.090039 17117 catalog_manager.cc:5484] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.093808 17264 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.094564 17791 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:43.099267 17397 consensus_queue.cc:237] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:43.100132 17398 consensus_queue.cc:237] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:43.101552 17925 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.102028 17925 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.102129 18058 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.102131 19907 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.102444 18057 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.102515 19017 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.102823 19017 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.103485 17660 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.103778 17264 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.103928 17529 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.104053 17265 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.103928 17530 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.104667 19894 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.104805 17662 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.104993 19894 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.105165 19188 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.105230 19891 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.105333 19892 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.105547 19891 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.106508 19891 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.106909 17793 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:43.107784 19017 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:43.108497 17791 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:43.108714 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138:33637): Couldn't send request to peer ab18888a9724446b9ee4fb9efe811b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8500b198108649a7bcb614c93cf53a3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:43.109113 19894 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.109674 19907 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.109813 19894 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.112849 17925 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.113155 18057 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.113441 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8500b198108649a7bcb614c93cf53a3a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:43.113756 17264 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.116362 17124 catalog_manager.cc:5484] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.116690 17662 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.117080 17529 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.117975 17660 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.118990 17791 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.114598 17265 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.116362 17118 catalog_manager.cc:5484] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.121290 18057 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.122996 17529 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:43.126024 17925 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.129451 17398 consensus_queue.cc:237] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:43.131249 17265 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.131712 19907 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.131796 17660 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.132167 17529 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:43.132278 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138:33637): Couldn't send request to peer ab18888a9724446b9ee4fb9efe811b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42be6135d7654954b6886c32dfc688c5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:43.132556 18057 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.132576 17793 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.132704 19188 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.133013 17925 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:43.133016 19188 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.132915 17791 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.133327 19188 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.133790 19892 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.135115 19907 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:43.138247 19188 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.138886 17791 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.140190 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 42be6135d7654954b6886c32dfc688c5 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:43.140420 17265 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.140882 17124 catalog_manager.cc:5484] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:43.141418 17925 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.142557 17529 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.145685 17660 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.148306 18057 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:43.203199 19682 heartbeater.cc:503] Master 127.0.99.190:35233 was elected leader, sending a full tablet report...
I20240503 08:00:43.353183 20016 ts_tablet_manager.cc:902] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:43.353883 20016 tablet_copy_client.cc:308] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33199
I20240503 08:00:43.358008 17286 tablet_copy_service.cc:140] P 8eb6822290b046f88c861159aa9421ec: Received BeginTabletCopySession request for tablet 99d8d89b8561420781b7e7a9b9c778a0 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:37849)
I20240503 08:00:43.358135 17286 tablet_copy_service.cc:161] P 8eb6822290b046f88c861159aa9421ec: Beginning new tablet copy session on tablet 99d8d89b8561420781b7e7a9b9c778a0 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:37849: session id = b65378a282b9443484843de0b8906df9-99d8d89b8561420781b7e7a9b9c778a0
I20240503 08:00:43.359197 17286 tablet_copy_source_session.cc:205] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:43.362413 20016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d8d89b8561420781b7e7a9b9c778a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:43.366468 20022 ts_tablet_manager.cc:902] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:43.367211 20022 tablet_copy_client.cc:308] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39647
I20240503 08:00:43.367887 20016 tablet_copy_client.cc:783] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:43.368067 20016 tablet_copy_client.cc:647] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:43.369705 20016 tablet_copy_client.cc:523] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:43.371270 20016 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:43.372027 17550 tablet_copy_service.cc:140] P 7af8f2a481294a2cb93e5761947978cc: Received BeginTabletCopySession request for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:40241)
I20240503 08:00:43.372161 17550 tablet_copy_service.cc:161] P 7af8f2a481294a2cb93e5761947978cc: Beginning new tablet copy session on tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:40241: session id = 0665f4f85b984a8bba213ad479a88bad-a79ddd7d381c4ccbbd1a1e7d3ec8366f
I20240503 08:00:43.373469 17550 tablet_copy_source_session.cc:205] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:43.376191 20022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:43.376708 20016 log.cc:824] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:43.382160 20022 tablet_copy_client.cc:783] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:43.382375 20022 tablet_copy_client.cc:647] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:43.383965 20022 tablet_copy_client.cc:523] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:43.385629 20022 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:43.391849 20022 log.cc:824] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:43.403136 20016 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:43.403470 20016 tablet_bootstrap.cc:492] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:43.403779 20016 ts_tablet_manager.cc:1367] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.000s
I20240503 08:00:43.405509 20034 ts_tablet_manager.cc:902] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:43.406225 20016 raft_consensus.cc:348] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:43.406476 20016 raft_consensus.cc:729] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:43.406821 20016 consensus_queue.cc:260] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:43.407646 20016 ts_tablet_manager.cc:1398] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 08:00:43.406329 20036 ts_tablet_manager.cc:902] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863)
I20240503 08:00:43.410087 17286 tablet_copy_service.cc:342] P 8eb6822290b046f88c861159aa9421ec: Request end of tablet copy session b65378a282b9443484843de0b8906df9-99d8d89b8561420781b7e7a9b9c778a0 received from {username='slave'} at 127.0.99.136:37849
I20240503 08:00:43.410185 17286 tablet_copy_service.cc:433] P 8eb6822290b046f88c861159aa9421ec: ending tablet copy session b65378a282b9443484843de0b8906df9-99d8d89b8561420781b7e7a9b9c778a0 on tablet 99d8d89b8561420781b7e7a9b9c778a0 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:43.410368 20036 tablet_copy_client.cc:308] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:34863
I20240503 08:00:43.412807 20034 tablet_copy_client.cc:308] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39647
I20240503 08:00:43.413838 17550 tablet_copy_service.cc:140] P 7af8f2a481294a2cb93e5761947978cc: Received BeginTabletCopySession request for tablet 143109561ad549d69a59992c02fe23af from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:40241)
I20240503 08:00:43.413935 17550 tablet_copy_service.cc:161] P 7af8f2a481294a2cb93e5761947978cc: Beginning new tablet copy session on tablet 143109561ad549d69a59992c02fe23af from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:40241: session id = 0665f4f85b984a8bba213ad479a88bad-143109561ad549d69a59992c02fe23af
I20240503 08:00:43.414830 17550 tablet_copy_source_session.cc:205] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:43.415298 17682 tablet_copy_service.cc:140] P 0ce150b975ff4040bedde6319d5cf3d5: Received BeginTabletCopySession request for tablet 882c64fe082243fbbb3099f2bf6e33e6 from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:55309)
I20240503 08:00:43.415439 17682 tablet_copy_service.cc:161] P 0ce150b975ff4040bedde6319d5cf3d5: Beginning new tablet copy session on tablet 882c64fe082243fbbb3099f2bf6e33e6 from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:55309: session id = 0665f4f85b984a8bba213ad479a88bad-882c64fe082243fbbb3099f2bf6e33e6
I20240503 08:00:43.415777 20034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143109561ad549d69a59992c02fe23af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:43.416915 17682 tablet_copy_source_session.cc:205] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:43.418737 20036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882c64fe082243fbbb3099f2bf6e33e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:43.420336 20051 ts_tablet_manager.cc:902] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: Initiating tablet copy from peer 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:43.421034 20051 tablet_copy_client.cc:308] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33199
I20240503 08:00:43.423194 20036 tablet_copy_client.cc:783] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:43.423506 20036 tablet_copy_client.cc:647] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:43.424576 20034 tablet_copy_client.cc:783] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:43.424746 20034 tablet_copy_client.cc:647] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:43.424600 20022 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:43.425041 20022 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:43.425451 20036 tablet_copy_client.cc:523] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:43.426311 20034 tablet_copy_client.cc:523] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:43.426904 20036 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:43.425493 20022 ts_tablet_manager.cc:1367] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.040s user 0.029s sys 0.004s
I20240503 08:00:43.428187 17286 tablet_copy_service.cc:140] P 8eb6822290b046f88c861159aa9421ec: Received BeginTabletCopySession request for tablet 8a03fdbe7f824674a351fd4598dc2c63 from peer ab18888a9724446b9ee4fb9efe811b09 ({username='slave'} at 127.0.99.138:34835)
I20240503 08:00:43.428287 17286 tablet_copy_service.cc:161] P 8eb6822290b046f88c861159aa9421ec: Beginning new tablet copy session on tablet 8a03fdbe7f824674a351fd4598dc2c63 from peer ab18888a9724446b9ee4fb9efe811b09 at {username='slave'} at 127.0.99.138:34835: session id = ab18888a9724446b9ee4fb9efe811b09-8a03fdbe7f824674a351fd4598dc2c63
I20240503 08:00:43.428748 20034 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:43.429205 17286 tablet_copy_source_session.cc:205] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:43.432417 20051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:43.432844 20022 raft_consensus.cc:348] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.433115 20022 raft_consensus.cc:729] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:43.433440 20022 consensus_queue.cc:260] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.434252 20022 ts_tablet_manager.cc:1398] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:00:43.435562 17550 tablet_copy_service.cc:342] P 7af8f2a481294a2cb93e5761947978cc: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-a79ddd7d381c4ccbbd1a1e7d3ec8366f received from {username='slave'} at 127.0.99.137:40241
I20240503 08:00:43.435652 17550 tablet_copy_service.cc:433] P 7af8f2a481294a2cb93e5761947978cc: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-a79ddd7d381c4ccbbd1a1e7d3ec8366f on tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:43.438745 20051 tablet_copy_client.cc:783] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:43.439085 20051 tablet_copy_client.cc:647] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:43.440516 20051 tablet_copy_client.cc:523] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:43.443123 20051 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap starting.
I20240503 08:00:43.446194 20022 ts_tablet_manager.cc:902] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:43.446583 20022 tablet_copy_client.cc:308] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:43.449195 20051 log.cc:824] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: Log is configured to *not* fsync() on all Append() calls
I20240503 08:00:43.451658 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet ed32a1099e044448aaa0c9c9f58f74e6 from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:50179)
I20240503 08:00:43.451792 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet ed32a1099e044448aaa0c9c9f58f74e6 from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:50179: session id = 0665f4f85b984a8bba213ad479a88bad-ed32a1099e044448aaa0c9c9f58f74e6
I20240503 08:00:43.453148 17418 tablet_copy_source_session.cc:205] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:43.454329 20022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32a1099e044448aaa0c9c9f58f74e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:43.456238 20022 tablet_copy_client.cc:783] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:43.456362 20022 tablet_copy_client.cc:647] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:43.458235 20022 tablet_copy_client.cc:523] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:43.459321 20022 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:43.462853 20036 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:43.463174 20036 tablet_bootstrap.cc:492] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:43.463366 20036 ts_tablet_manager.cc:1367] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.037s user 0.025s sys 0.000s
I20240503 08:00:43.463894 20036 raft_consensus.cc:348] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.464031 20036 raft_consensus.cc:729] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:43.464161 20036 consensus_queue.cc:260] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.464442 20036 ts_tablet_manager.cc:1398] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:43.466045 17682 tablet_copy_service.cc:342] P 0ce150b975ff4040bedde6319d5cf3d5: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-882c64fe082243fbbb3099f2bf6e33e6 received from {username='slave'} at 127.0.99.137:55309
I20240503 08:00:43.466137 17682 tablet_copy_service.cc:433] P 0ce150b975ff4040bedde6319d5cf3d5: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-882c64fe082243fbbb3099f2bf6e33e6 on tablet 882c64fe082243fbbb3099f2bf6e33e6 with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:43.469159 20034 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:43.469441 20034 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:43.469631 20034 ts_tablet_manager.cc:1367] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.041s user 0.026s sys 0.000s
I20240503 08:00:43.470052 20034 raft_consensus.cc:348] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.470155 20034 raft_consensus.cc:729] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:43.470254 20034 consensus_queue.cc:260] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.470517 20034 ts_tablet_manager.cc:1398] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:43.470942 17550 tablet_copy_service.cc:342] P 7af8f2a481294a2cb93e5761947978cc: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-143109561ad549d69a59992c02fe23af received from {username='slave'} at 127.0.99.137:40241
I20240503 08:00:43.471030 17550 tablet_copy_service.cc:433] P 7af8f2a481294a2cb93e5761947978cc: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-143109561ad549d69a59992c02fe23af on tablet 143109561ad549d69a59992c02fe23af with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:43.487201 20051 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:43.487651 20051 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap complete.
I20240503 08:00:43.487977 20051 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: Time spent bootstrapping tablet: real 0.045s user 0.026s sys 0.005s
I20240503 08:00:43.490558 20051 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:43.490882 20051 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab18888a9724446b9ee4fb9efe811b09, State: Initialized, Role: LEARNER
I20240503 08:00:43.491241 20051 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:43.492148 20051 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:00:43.492761 17286 tablet_copy_service.cc:342] P 8eb6822290b046f88c861159aa9421ec: Request end of tablet copy session ab18888a9724446b9ee4fb9efe811b09-8a03fdbe7f824674a351fd4598dc2c63 received from {username='slave'} at 127.0.99.138:34835
I20240503 08:00:43.492835 17286 tablet_copy_service.cc:433] P 8eb6822290b046f88c861159aa9421ec: ending tablet copy session ab18888a9724446b9ee4fb9efe811b09-8a03fdbe7f824674a351fd4598dc2c63 on tablet 8a03fdbe7f824674a351fd4598dc2c63 with peer ab18888a9724446b9ee4fb9efe811b09
I20240503 08:00:43.503041 20022 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:43.503263 20022 tablet_bootstrap.cc:492] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:43.503414 20022 ts_tablet_manager.cc:1367] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.000s
I20240503 08:00:43.503791 20022 raft_consensus.cc:348] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.503926 20022 raft_consensus.cc:729] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:43.504015 20022 consensus_queue.cc:260] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.504372 20022 ts_tablet_manager.cc:1398] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:00:43.505026 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-ed32a1099e044448aaa0c9c9f58f74e6 received from {username='slave'} at 127.0.99.137:50179
I20240503 08:00:43.505149 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-ed32a1099e044448aaa0c9c9f58f74e6 on tablet ed32a1099e044448aaa0c9c9f58f74e6 with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:43.541770 20022 ts_tablet_manager.cc:902] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:43.542191 20022 tablet_copy_client.cc:308] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:43.545318 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet 70f073cc41d54638a2c51746b94a5ece from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:47395)
I20240503 08:00:43.545475 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet 70f073cc41d54638a2c51746b94a5ece from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:47395: session id = 0665f4f85b984a8bba213ad479a88bad-70f073cc41d54638a2c51746b94a5ece
I20240503 08:00:43.546717 17946 tablet_copy_source_session.cc:205] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:43.547864 20022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f073cc41d54638a2c51746b94a5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:43.549769 20022 tablet_copy_client.cc:783] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:43.549912 20022 tablet_copy_client.cc:647] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:43.551153 20022 tablet_copy_client.cc:523] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:43.552093 20022 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:43.568027 20051 ts_tablet_manager.cc:902] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:43.568467 20051 tablet_copy_client.cc:308] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:43.569459 20022 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:43.569679 20022 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:43.569829 20022 ts_tablet_manager.cc:1367] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.004s
I20240503 08:00:43.570199 20022 raft_consensus.cc:348] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.570334 20022 raft_consensus.cc:729] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:43.570442 20022 consensus_queue.cc:260] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:43.571122 20022 ts_tablet_manager.cc:1398] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:43.572206 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet 8500b198108649a7bcb614c93cf53a3a from peer ab18888a9724446b9ee4fb9efe811b09 ({username='slave'} at 127.0.99.138:48513)
I20240503 08:00:43.572199 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-70f073cc41d54638a2c51746b94a5ece received from {username='slave'} at 127.0.99.137:47395
I20240503 08:00:43.572296 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-70f073cc41d54638a2c51746b94a5ece on tablet 70f073cc41d54638a2c51746b94a5ece with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:43.572304 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet 8500b198108649a7bcb614c93cf53a3a from peer ab18888a9724446b9ee4fb9efe811b09 at {username='slave'} at 127.0.99.138:48513: session id = ab18888a9724446b9ee4fb9efe811b09-8500b198108649a7bcb614c93cf53a3a
I20240503 08:00:43.573132 17418 tablet_copy_source_session.cc:205] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:43.573921 20051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:43.575821 20051 tablet_copy_client.cc:783] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:43.575946 20051 tablet_copy_client.cc:647] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:43.576880 20051 tablet_copy_client.cc:523] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:43.578037 20051 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: Bootstrap starting.
I20240503 08:00:43.596386 20051 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:43.596612 20051 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: Bootstrap complete.
I20240503 08:00:43.596771 20051 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.001s
I20240503 08:00:43.597295 20051 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:43.597427 20051 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab18888a9724446b9ee4fb9efe811b09, State: Initialized, Role: LEARNER
I20240503 08:00:43.597527 20051 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:43.597805 20051 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 08:00:43.598295 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session ab18888a9724446b9ee4fb9efe811b09-8500b198108649a7bcb614c93cf53a3a received from {username='slave'} at 127.0.99.138:48513
I20240503 08:00:43.598374 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session ab18888a9724446b9ee4fb9efe811b09-8500b198108649a7bcb614c93cf53a3a on tablet 8500b198108649a7bcb614c93cf53a3a with peer ab18888a9724446b9ee4fb9efe811b09
I20240503 08:00:43.625660 20051 ts_tablet_manager.cc:902] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:43.626083 20051 tablet_copy_client.cc:308] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:43.626452 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet 42be6135d7654954b6886c32dfc688c5 from peer ab18888a9724446b9ee4fb9efe811b09 ({username='slave'} at 127.0.99.138:48513)
I20240503 08:00:43.626546 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet 42be6135d7654954b6886c32dfc688c5 from peer ab18888a9724446b9ee4fb9efe811b09 at {username='slave'} at 127.0.99.138:48513: session id = ab18888a9724446b9ee4fb9efe811b09-42be6135d7654954b6886c32dfc688c5
I20240503 08:00:43.627379 17418 tablet_copy_source_session.cc:205] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:43.628109 20051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be6135d7654954b6886c32dfc688c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:43.630048 20051 tablet_copy_client.cc:783] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:43.630179 20051 tablet_copy_client.cc:647] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:43.632491 20051 tablet_copy_client.cc:523] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:43.633431 20051 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap starting.
I20240503 08:00:43.649662 20051 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:43.649935 20051 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap complete.
I20240503 08:00:43.650087 20051 ts_tablet_manager.cc:1367] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.004s
I20240503 08:00:43.650419 20051 raft_consensus.cc:348] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:43.650540 20051 raft_consensus.cc:729] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab18888a9724446b9ee4fb9efe811b09, State: Initialized, Role: LEARNER
I20240503 08:00:43.650630 20051 consensus_queue.cc:260] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:43.650885 20051 ts_tablet_manager.cc:1398] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:43.651293 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session ab18888a9724446b9ee4fb9efe811b09-42be6135d7654954b6886c32dfc688c5 received from {username='slave'} at 127.0.99.138:48513
I20240503 08:00:43.651368 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session ab18888a9724446b9ee4fb9efe811b09-42be6135d7654954b6886c32dfc688c5 on tablet 42be6135d7654954b6886c32dfc688c5 with peer ab18888a9724446b9ee4fb9efe811b09
I20240503 08:00:43.796897 19374 raft_consensus.cc:1203] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:43.831880 19374 raft_consensus.cc:1203] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:43.842574 19505 raft_consensus.cc:1203] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:43.910285 19374 raft_consensus.cc:1203] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:43.934141 19374 raft_consensus.cc:1203] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:43.937610 19636 raft_consensus.cc:1203] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:44.008561 19374 raft_consensus.cc:1203] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:44.173900 19505 raft_consensus.cc:1203] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:44.178386 19505 raft_consensus.cc:1203] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:44.202317 19856 raft_consensus.cc:1050] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:44.202831 19856 consensus_queue.cc:237] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.204653 17398 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.204704 19374 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.204991 18057 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.205516 17925 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.205498 19856 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.205884 19856 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.206043 19855 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.206370 19855 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.207393 17264 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.207741 17662 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.208125 20220 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.208330 20162 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.209035 17793 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.209743 20162 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.210800 20162 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.212515 17662 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.212553 18057 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.212751 17264 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.214339 17397 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.214339 17925 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.215169 17123 catalog_manager.cc:5484] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.216161 17793 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.218358 19374 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.224249 17529 consensus_queue.cc:237] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.226122 19374 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.226122 18057 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.226153 17793 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.226266 17397 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.226403 17925 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.226495 19856 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.226665 20233 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.226787 17264 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.226840 20162 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.227244 20162 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.227366 19856 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.227459 20220 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.230648 17925 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.230671 17264 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.232863 20162 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.233085 17118 catalog_manager.cc:5484] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.233160 17397 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.234710 19374 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.235641 18057 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.235893 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:44.236059 17640 tablet_service.cc:1514] Processing DeleteTablet for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f with delete_type TABLET_DATA_TOMBSTONED (TS 0ce150b975ff4040bedde6319d5cf3d5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35622
I20240503 08:00:44.236270 17793 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.237912 20251 tablet_replica.cc:330] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5: stopping tablet replica
I20240503 08:00:44.238279 20251 raft_consensus.cc:2229] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.239567 20251 raft_consensus.cc:2258] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.240957 20251 ts_tablet_manager.cc:1875] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:44.243147 20251 ts_tablet_manager.cc:1888] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:44.243223 20251 log.cc:1196] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0ce150b975ff4040bedde6319d5cf3d5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a79ddd7d381c4ccbbd1a1e7d3ec8366f
I20240503 08:00:44.243692 17108 catalog_manager.cc:4830] TS 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863): tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f (table rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]) successfully deleted
I20240503 08:00:44.253234 20164 raft_consensus.cc:1050] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: attempting to promote NON_VOTER ab18888a9724446b9ee4fb9efe811b09 to VOTER
I20240503 08:00:44.253780 20164 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:44.255668 17662 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.255668 17793 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.255806 17529 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.256093 19505 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.256137 19133 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.256417 17397 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.256870 18057 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.256902 17925 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.256928 20163 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.257272 20163 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.257397 19133 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.257711 19133 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.258015 19133 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.259721 20263 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.260993 20265 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.261889 17529 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.261862 18057 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.263377 17662 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.263950 17793 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.264217 17119 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.266064 17397 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.267581 17925 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.270303 19505 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.274272 17264 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:44.275712 18057 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.275822 17793 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.276211 17529 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.276389 17662 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.276522 20163 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.276803 19505 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.276909 20164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.278384 20264 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.278550 19133 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.278820 20263 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.279911 17397 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.280252 20164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.280747 20263 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.281208 19505 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.281209 17529 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.282701 18057 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.283717 17118 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 reported cstate change: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.283980 17791 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.284425 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a03fdbe7f824674a351fd4598dc2c63 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:44.285410 17662 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.285840 17397 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.287290 17905 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a03fdbe7f824674a351fd4598dc2c63 with delete_type TABLET_DATA_TOMBSTONED (TS 78e52cf3c8fa435bb5ca36061227ac41 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34568
I20240503 08:00:44.289599 20284 tablet_replica.cc:330] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41: stopping tablet replica
I20240503 08:00:44.289923 20284 raft_consensus.cc:2229] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.290228 20284 raft_consensus.cc:2258] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.291438 20284 ts_tablet_manager.cc:1875] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:44.293146 20284 ts_tablet_manager.cc:1888] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:44.293233 20284 log.cc:1196] T 8a03fdbe7f824674a351fd4598dc2c63 P 78e52cf3c8fa435bb5ca36061227ac41: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/8a03fdbe7f824674a351fd4598dc2c63
I20240503 08:00:44.293821 17109 catalog_manager.cc:4830] TS 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175): tablet 8a03fdbe7f824674a351fd4598dc2c63 (table rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]) successfully deleted
I20240503 08:00:44.313872 20249 raft_consensus.cc:1050] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:44.314368 20249 consensus_queue.cc:237] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.315965 17791 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.316031 19374 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.316347 20162 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.316718 18057 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.316741 20162 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.317332 17264 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.317406 17662 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.317788 20249 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.317939 19856 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.318253 19856 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.318481 17397 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.318866 17925 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.318917 20249 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.319196 19856 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.321305 20249 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.321704 17662 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.322453 17791 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.323125 17264 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.323266 19374 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.324580 17124 catalog_manager.cc:5484] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:44.325960 17925 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.326774 18057 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.327119 17397 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.332087 17529 consensus_queue.cc:237] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.333420 19374 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.333725 17397 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.333833 17264 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.333966 18057 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.333966 17925 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.334010 19855 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.334223 19856 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.334331 19855 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.334394 17791 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.335110 19855 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.335270 20243 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.335613 20233 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.337245 19855 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.337612 17264 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.339018 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 143109561ad549d69a59992c02fe23af with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:44.339133 17397 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.339586 17791 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.340245 19374 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.340896 17123 catalog_manager.cc:5484] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.343020 18057 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.343815 17925 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.344666 17640 tablet_service.cc:1514] Processing DeleteTablet for tablet 143109561ad549d69a59992c02fe23af with delete_type TABLET_DATA_TOMBSTONED (TS 0ce150b975ff4040bedde6319d5cf3d5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35622
I20240503 08:00:44.344872 20251 tablet_replica.cc:330] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: stopping tablet replica
I20240503 08:00:44.345014 20251 raft_consensus.cc:2229] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.345239 20251 raft_consensus.cc:2258] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.345897 20251 ts_tablet_manager.cc:1875] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:44.347841 20251 ts_tablet_manager.cc:1888] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:44.347898 20251 log.cc:1196] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/143109561ad549d69a59992c02fe23af
I20240503 08:00:44.348281 17108 catalog_manager.cc:4830] TS 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863): tablet 143109561ad549d69a59992c02fe23af (table rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]) successfully deleted
I20240503 08:00:44.371732 19892 raft_consensus.cc:1050] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:44.372254 19892 consensus_queue.cc:237] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.374157 17925 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.374167 17791 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.374267 17529 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.374347 17264 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.374446 17662 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.374781 18057 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.374842 19374 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.374768 20301 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.375012 19892 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.375149 19964 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.375311 20300 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.375468 19907 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.375646 20299 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.376986 19964 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.379438 20299 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.379809 19374 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.379866 17529 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.379871 18057 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.381863 17791 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.382663 17121 catalog_manager.cc:5484] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad reported cstate change: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.383702 17925 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.384003 17265 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.385273 17662 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.390758 17397 consensus_queue.cc:237] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.392149 19374 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.392270 18057 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.392510 19907 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.392581 17529 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.392666 17925 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.392926 19964 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.393082 19907 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.393121 17662 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.393342 17265 raft_consensus.cc:1261] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.393384 19907 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.393553 20302 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.393815 19892 consensus_queue.cc:1035] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.398945 19964 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.400014 17925 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.400007 17529 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.400667 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ed32a1099e044448aaa0c9c9f58f74e6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:44.400650 19374 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.401283 17662 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.403139 17119 catalog_manager.cc:5484] T ed32a1099e044448aaa0c9c9f58f74e6 P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.403566 17265 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.403708 18057 raft_consensus.cc:2932] T ed32a1099e044448aaa0c9c9f58f74e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.406808 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet ed32a1099e044448aaa0c9c9f58f74e6 with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:44.407323 20310 tablet_replica.cc:330] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:44.407585 20310 raft_consensus.cc:2229] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.407905 20310 raft_consensus.cc:2258] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.410138 20310 ts_tablet_manager.cc:1875] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:44.411973 20310 ts_tablet_manager.cc:1888] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:44.412042 20310 log.cc:1196] T ed32a1099e044448aaa0c9c9f58f74e6 P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ed32a1099e044448aaa0c9c9f58f74e6
I20240503 08:00:44.412477 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet ed32a1099e044448aaa0c9c9f58f74e6 (table rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]) successfully deleted
I20240503 08:00:44.413523 20165 raft_consensus.cc:1050] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:44.413921 20165 consensus_queue.cc:237] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.415544 19374 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.415625 18057 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.415840 17791 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.416028 17265 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.416040 17530 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.416106 17397 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.416545 20313 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.416669 17662 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.416787 20312 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.417101 20312 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.417269 20314 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.417420 19948 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.417639 20316 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.418032 19948 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.420322 20165 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.420722 18057 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.421111 17397 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.421725 17265 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.423141 19374 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.423354 17662 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.423992 17530 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.424223 17121 catalog_manager.cc:5484] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:44.428045 17791 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.428400 17925 consensus_queue.cc:237] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.429922 17529 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.430053 17397 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.430657 18057 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.430706 19374 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.430747 17662 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.430887 17265 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.430976 19948 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.431140 20312 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.431452 20312 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.431594 20313 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.432006 20312 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.432487 19948 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.434307 19948 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.434604 17530 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.435074 19374 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.435559 17109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 70f073cc41d54638a2c51746b94a5ece with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:44.436342 18057 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.437209 17124 catalog_manager.cc:5484] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.437390 17398 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.438239 17265 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.438794 17662 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.439709 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet 70f073cc41d54638a2c51746b94a5ece with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:44.442195 20310 tablet_replica.cc:330] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:44.442333 20310 raft_consensus.cc:2229] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.442503 20310 raft_consensus.cc:2258] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.443104 20310 ts_tablet_manager.cc:1875] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:44.444528 20310 ts_tablet_manager.cc:1888] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:44.444587 20310 log.cc:1196] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/70f073cc41d54638a2c51746b94a5ece
I20240503 08:00:44.444967 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet 70f073cc41d54638a2c51746b94a5ece (table rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]) successfully deleted
I20240503 08:00:44.463373 19874 raft_consensus.cc:1050] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:44.463892 19874 consensus_queue.cc:237] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.465718 19374 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.465859 17265 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.465983 17791 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.466183 20207 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.466318 19874 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.466745 17530 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.467058 20213 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.467504 17925 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.467639 17398 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.467828 20213 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.468096 20213 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.468549 20207 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.469866 18057 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.470162 20213 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.471184 20213 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.471848 17398 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.472318 17530 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.473256 17925 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.473410 17791 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.474436 19374 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.475271 17123 catalog_manager.cc:5484] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:44.478667 17265 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.480329 18057 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.483630 17662 consensus_queue.cc:237] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:44.484892 18057 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.484895 17530 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.485031 17791 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.485281 20327 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.485450 20207 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.485622 17398 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.485626 19374 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.485693 17265 raft_consensus.cc:1261] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.485939 20207 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.486169 19874 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.486280 20207 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.486508 20328 consensus_queue.cc:1035] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.488197 20207 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.488968 17530 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.489272 19374 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.490007 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 882c64fe082243fbbb3099f2bf6e33e6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:44.490095 17265 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.490581 17791 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.491755 17124 catalog_manager.cc:5484] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:44.492043 18057 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.492945 17398 raft_consensus.cc:2932] T 882c64fe082243fbbb3099f2bf6e33e6 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:44.494396 17905 tablet_service.cc:1514] Processing DeleteTablet for tablet 882c64fe082243fbbb3099f2bf6e33e6 with delete_type TABLET_DATA_TOMBSTONED (TS 78e52cf3c8fa435bb5ca36061227ac41 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34568
I20240503 08:00:44.494611 20284 tablet_replica.cc:330] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41: stopping tablet replica
I20240503 08:00:44.494727 20284 raft_consensus.cc:2229] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.494920 20284 raft_consensus.cc:2258] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.496245 20284 ts_tablet_manager.cc:1875] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:44.498135 20284 ts_tablet_manager.cc:1888] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:44.498195 20284 log.cc:1196] T 882c64fe082243fbbb3099f2bf6e33e6 P 78e52cf3c8fa435bb5ca36061227ac41: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/882c64fe082243fbbb3099f2bf6e33e6
I20240503 08:00:44.498600 17109 catalog_manager.cc:4830] TS 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175): tablet 882c64fe082243fbbb3099f2bf6e33e6 (table rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]) successfully deleted
I20240503 08:00:44.547230 20263 raft_consensus.cc:1050] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:44.547674 20263 consensus_queue.cc:237] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:44.549527 18057 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.549549 17925 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.549890 17398 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.549901 17791 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.549912 20382 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.550285 17530 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.550442 20265 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.550676 19636 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.550776 17662 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.551084 20265 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.551353 20265 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.551517 20163 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.551928 20163 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.552551 20266 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.555462 20264 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.556070 19636 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.556329 18057 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.556507 17529 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.557986 17925 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.558406 17398 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.558624 17662 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.560072 17121 catalog_manager.cc:5484] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.562188 17791 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.565773 17265 consensus_queue.cc:237] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:44.567128 19636 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.567184 17925 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.567184 18057 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.567255 17398 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.567488 20265 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.567633 19133 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.567817 20164 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.567934 20266 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.568019 17529 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.568390 17662 raft_consensus.cc:1261] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.568389 19133 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.568830 19133 consensus_queue.cc:1035] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.570168 19133 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.570585 17398 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.570811 17925 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.571568 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99d8d89b8561420781b7e7a9b9c778a0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:44.572540 17124 catalog_manager.cc:5484] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:44.572618 18057 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.572700 17529 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.574200 17662 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.576167 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d8d89b8561420781b7e7a9b9c778a0 with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:44.576586 20310 tablet_replica.cc:330] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:44.576718 20310 raft_consensus.cc:2229] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.576864 20310 raft_consensus.cc:2258] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.577553 20310 ts_tablet_manager.cc:1875] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:44.577606 19636 raft_consensus.cc:2932] T 99d8d89b8561420781b7e7a9b9c778a0 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:44.579591 20310 ts_tablet_manager.cc:1888] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:44.579665 20310 log.cc:1196] T 99d8d89b8561420781b7e7a9b9c778a0 P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/99d8d89b8561420781b7e7a9b9c778a0
I20240503 08:00:44.580140 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet 99d8d89b8561420781b7e7a9b9c778a0 (table rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]) successfully deleted
I20240503 08:00:44.698923 20302 raft_consensus.cc:1050] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: attempting to promote NON_VOTER ab18888a9724446b9ee4fb9efe811b09 to VOTER
I20240503 08:00:44.699388 20302 consensus_queue.cc:237] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:44.701206 17662 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.701277 17925 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.701449 17529 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.701575 17265 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.701776 19505 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.701756 20300 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.701920 19964 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.702167 19907 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.702428 20300 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.702615 17791 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.702723 20299 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.703760 19964 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.705572 18057 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.705613 19964 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.705926 17791 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.706089 19907 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.706385 17265 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.708335 17925 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.708828 17120 catalog_manager.cc:5484] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:44.711201 19505 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.712790 17529 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.713955 17662 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.715196 18057 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.740032 19892 raft_consensus.cc:1050] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: attempting to promote NON_VOTER ab18888a9724446b9ee4fb9efe811b09 to VOTER
I20240503 08:00:44.740460 19892 consensus_queue.cc:237] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:44.742384 17265 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.742379 17791 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.742430 17529 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.742750 20299 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.742939 19964 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.743073 17925 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.743101 20300 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.743398 19505 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.743525 19964 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.743718 17662 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.744252 18057 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:44.744925 20300 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.745293 20300 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.746315 20299 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:44.747222 20299 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.748209 19505 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.749169 17791 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.749308 17925 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.751451 17124 catalog_manager.cc:5484] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 reported cstate change: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.751742 17265 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.751883 18057 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.751935 17529 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.755618 17662 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.759310 17398 consensus_queue.cc:237] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:44.759668 17662 consensus_queue.cc:237] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:44.760869 18057 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.761122 17791 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.761161 19505 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.761457 17265 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.761535 17529 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.761535 17398 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.761831 20207 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.761821 20302 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.761988 19964 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.762221 20326 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.762276 19964 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.763139 18058 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.763455 17924 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.763545 20328 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.764001 20207 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.764400 20300 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.764683 17266 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.765108 17925 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.765383 17791 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.765986 20328 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.766950 17529 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:44.767051 20328 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.767228 20207 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.767374 19964 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.767681 19964 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:44.768978 17529 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.770099 17398 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.770112 17924 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.770494 17118 catalog_manager.cc:5484] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:44.771404 18058 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.774806 20300 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.775702 19505 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.775827 17662 consensus_queue.cc:237] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:44.776742 17925 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.777726 17118 catalog_manager.cc:5484] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:44.778882 17794 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.779345 17266 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.779676 17398 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.780187 20326 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.780367 18058 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.780447 20326 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.781553 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8500b198108649a7bcb614c93cf53a3a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:44.781766 20207 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.781850 17640 tablet_service.cc:1514] Processing DeleteTablet for tablet 8500b198108649a7bcb614c93cf53a3a with delete_type TABLET_DATA_TOMBSTONED (TS 0ce150b975ff4040bedde6319d5cf3d5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35622
I20240503 08:00:44.782101 20251 tablet_replica.cc:330] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5: stopping tablet replica
I20240503 08:00:44.782227 20251 raft_consensus.cc:2229] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:44.782371 20251 raft_consensus.cc:2258] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:44.782786 17530 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.783295 18057 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.785598 20251 ts_tablet_manager.cc:1875] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:44.786000 17925 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.787050 17793 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.787451 20207 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.787573 20251 ts_tablet_manager.cc:1888] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:44.787633 20251 log.cc:1196] T 8500b198108649a7bcb614c93cf53a3a P 0ce150b975ff4040bedde6319d5cf3d5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8500b198108649a7bcb614c93cf53a3a
I20240503 08:00:44.788051 17108 catalog_manager.cc:4830] TS 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863): tablet 8500b198108649a7bcb614c93cf53a3a (table rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]) successfully deleted
I20240503 08:00:44.788157 17793 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.788751 17266 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:44.792466 20207 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.793280 17529 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.794028 20328 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.794847 20328 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.795181 17925 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.796124 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecc1f8b69268429f89cffb380e45e020 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:44.796591 17398 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.797101 18057 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.797552 17123 catalog_manager.cc:5484] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.797046 17793 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.799640 17530 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.802755 17266 raft_consensus.cc:1203] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.5] Dedup: 1.4->[1.5-1.5]
I20240503 08:00:44.802932 17266 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.809412 17530 consensus_queue.cc:237] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:44.811290 17265 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.811458 18057 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.811580 17398 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.811877 19855 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.812023 17925 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.812335 19855 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.812460 20243 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.812739 20243 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.813532 17793 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.813894 19856 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.814841 19856 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.815023 17266 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.816596 17266 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.816977 17925 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
W20240503 08:00:44.818594 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecc1f8b69268429f89cffb380e45e020. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:44.819274 17123 catalog_manager.cc:5484] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } }
I20240503 08:00:44.819883 17662 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.819839 18057 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.820245 17398 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.823472 17791 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.823881 20220 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.824925 17662 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.825583 17530 consensus_queue.cc:237] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:44.828037 17925 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.828091 18058 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.828595 19855 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.829018 17791 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.829104 20249 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.829555 20249 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.831605 17266 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.831941 17398 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.832034 18058 consensus_queue.cc:237] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } }
I20240503 08:00:44.832572 20249 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.832695 20243 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.833695 17397 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.834043 20249 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.834208 20442 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.834318 17530 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.834425 17265 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.834693 17661 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.834887 20442 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.835352 17925 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.835660 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5154e7f64d1542979f9363a429fa469c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:44.835769 17793 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.835845 20442 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.835985 19806 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.835922 17925 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.836329 19806 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.836417 17791 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.838268 19806 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.838454 17120 catalog_manager.cc:5484] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.840667 18058 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.841954 17266 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.846278 17398 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.847188 17660 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.847272 17530 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } }
I20240503 08:00:44.847898 20162 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.848814 17660 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.849292 19806 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } }
W20240503 08:00:44.850549 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5154e7f64d1542979f9363a429fa469c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:44.851336 17791 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } }
I20240503 08:00:44.851665 17924 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } }
I20240503 08:00:44.851768 17120 catalog_manager.cc:5484] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:44.854954 17266 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } }
I20240503 08:00:44.855530 17661 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } }
I20240503 08:00:44.859241 17397 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } }
I20240503 08:00:44.860208 18058 consensus_queue.cc:237] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:44.862193 17661 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.863260 17529 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.863297 19806 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.863493 17791 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.863672 17266 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.864043 20443 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.864180 19806 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.864287 20442 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.865613 17397 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.866032 20445 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.866631 20443 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.867362 17924 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.867606 17266 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.867975 19806 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:00:44.868640 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08e9b83cd82f4d82ae552501be742896. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:44.868599 17529 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.870572 17119 catalog_manager.cc:5484] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.870652 17791 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.871667 17109 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 08e9b83cd82f4d82ae552501be742896 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:44.871759 17661 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.872560 17397 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.872989 17924 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:44.880316 17791 consensus_queue.cc:237] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:44.881697 17529 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.881748 17397 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.882099 17266 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.882119 19871 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.882229 17924 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.882422 19871 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.882475 17661 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.882656 20463 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.882939 18058 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.882941 20463 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.883237 20463 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.884382 19871 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.885437 20461 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.885780 17266 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.887028 17661 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.888093 17118 catalog_manager.cc:5484] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "84305d92b082406380b107114e4cd61b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.889436 17397 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.890127 17924 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.893081 18058 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.894923 17791 consensus_queue.cc:237] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:44.895311 17529 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:44.897220 18058 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.897524 17266 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.897943 17924 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.898017 17529 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.898380 20462 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.898524 19871 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.898625 20460 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.898888 20460 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.899361 17924 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:44.900696 17661 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.900981 18058 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.901225 17791 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.901432 17397 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.901428 17530 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.901449 17661 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.902156 20472 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.902339 20462 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.902856 19948 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.903241 20165 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.903273 17266 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.903596 20315 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.903892 17397 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:44.903893 17529 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.904059 20313 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.904181 17661 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.906473 20315 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:44.906654 17124 catalog_manager.cc:5484] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "84305d92b082406380b107114e4cd61b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.906774 20315 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:00:44.908970 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138:33637): Couldn't send request to peer ab18888a9724446b9ee4fb9efe811b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbe0edacc8ce4ba8be5e533c82ea92e0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:44.909291 17398 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.909992 20460 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.910121 17265 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.912012 20313 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:44.912995 17266 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:44.913967 17791 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:44.914247 17110 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:44.914330 17397 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:44.914537 17660 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:44.914681 17124 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:44.915704 17530 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:44.917718 18057 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.920136 18058 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:44.920996 17924 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:44.924154 17266 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.924211 17925 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.924461 17397 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.925489 20316 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.925770 20316 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.926863 18058 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.926980 17791 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.927156 17530 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.927191 20316 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.927404 20314 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.927644 20314 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.929045 17660 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:44.929981 20316 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:44.930233 20315 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.930797 17791 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.931514 17397 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
W20240503 08:00:44.931756 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138:33637): Couldn't send request to peer ab18888a9724446b9ee4fb9efe811b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e57bf40798824baf8050988874664154. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:44.932312 17109 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e57bf40798824baf8050988874664154 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:44.932296 18058 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.932857 17119 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:44.934695 17266 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.937404 17660 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:44.937404 17530 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:45.153898 20528 ts_tablet_manager.cc:902] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863)
I20240503 08:00:45.154418 20528 tablet_copy_client.cc:308] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:34863
I20240503 08:00:45.158313 17682 tablet_copy_service.cc:140] P 0ce150b975ff4040bedde6319d5cf3d5: Received BeginTabletCopySession request for tablet ecc1f8b69268429f89cffb380e45e020 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:49971)
I20240503 08:00:45.158421 17682 tablet_copy_service.cc:161] P 0ce150b975ff4040bedde6319d5cf3d5: Beginning new tablet copy session on tablet ecc1f8b69268429f89cffb380e45e020 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:49971: session id = b65378a282b9443484843de0b8906df9-ecc1f8b69268429f89cffb380e45e020
I20240503 08:00:45.159080 17682 tablet_copy_source_session.cc:205] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:45.159819 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc1f8b69268429f89cffb380e45e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:45.161552 20528 tablet_copy_client.cc:783] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:45.161726 20528 tablet_copy_client.cc:647] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:45.162731 20528 tablet_copy_client.cc:523] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:45.163656 20528 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:45.174813 17397 consensus_queue.cc:237] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:45.176244 19505 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:45.176441 17791 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:45.176467 18058 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:45.176720 20299 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:45.176923 19964 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:45.177327 19964 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:45.177454 17660 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:45.178017 17530 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:45.179255 20299 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:45.179478 17266 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:45.179538 20299 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:45.179996 20299 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:45.180362 19907 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:45.180768 18058 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:45.182346 19505 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:45.183281 17120 catalog_manager.cc:5484] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:45.183964 17791 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:45.184188 20528 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:45.184418 20528 tablet_bootstrap.cc:492] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:45.185148 20528 ts_tablet_manager.cc:1367] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.007s
I20240503 08:00:45.185694 20528 raft_consensus.cc:348] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:45.185814 20528 raft_consensus.cc:729] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:45.185909 20528 consensus_queue.cc:260] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:45.186396 20528 ts_tablet_manager.cc:1398] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:45.186856 17905 tablet_service.cc:1514] Processing DeleteTablet for tablet 42be6135d7654954b6886c32dfc688c5 with delete_type TABLET_DATA_TOMBSTONED (TS 78e52cf3c8fa435bb5ca36061227ac41 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34568
I20240503 08:00:45.187376 20536 tablet_replica.cc:330] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41: stopping tablet replica
I20240503 08:00:45.187561 20536 raft_consensus.cc:2229] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:45.187803 20536 raft_consensus.cc:2258] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:45.188376 20536 ts_tablet_manager.cc:1875] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:45.188511 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 42be6135d7654954b6886c32dfc688c5 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:45.189438 17682 tablet_copy_service.cc:342] P 0ce150b975ff4040bedde6319d5cf3d5: Request end of tablet copy session b65378a282b9443484843de0b8906df9-ecc1f8b69268429f89cffb380e45e020 received from {username='slave'} at 127.0.99.136:49971
I20240503 08:00:45.189512 17682 tablet_copy_service.cc:433] P 0ce150b975ff4040bedde6319d5cf3d5: ending tablet copy session b65378a282b9443484843de0b8906df9-ecc1f8b69268429f89cffb380e45e020 on tablet ecc1f8b69268429f89cffb380e45e020 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:45.189641 17660 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:45.189713 17530 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:45.190727 20536 ts_tablet_manager.cc:1888] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:45.190788 20536 log.cc:1196] T 42be6135d7654954b6886c32dfc688c5 P 78e52cf3c8fa435bb5ca36061227ac41: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/42be6135d7654954b6886c32dfc688c5
I20240503 08:00:45.191288 17109 catalog_manager.cc:4830] TS 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175): tablet 42be6135d7654954b6886c32dfc688c5 (table rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]) successfully deleted
I20240503 08:00:45.196233 17660 consensus_queue.cc:237] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:45.197978 17397 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.198734 17530 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.199169 20213 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.199412 18058 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.199558 17924 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.199798 17791 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.200140 20213 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.200445 19874 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.200654 20328 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.200758 17265 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.200819 19874 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.201017 17266 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:45.204200 20328 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.204511 20328 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:45.204855 17397 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:45.206226 18058 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:45.208220 17118 catalog_manager.cc:5484] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:45.209452 17791 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:45.210974 17529 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:45.212241 17660 consensus_queue.cc:237] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:45.213235 17266 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
W20240503 08:00:45.214236 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba0615c2fbc840c7bbdfda22caf45de2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:45.215108 17924 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.215304 17397 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.215461 18058 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.215622 20326 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.215922 17791 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.216298 17924 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:45.216917 17529 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.216908 20326 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.217264 17266 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.218163 20326 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.218266 20213 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.219167 20325 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.219440 20325 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.220674 20213 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:45.222329 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ba0615c2fbc840c7bbdfda22caf45de2 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:45.222921 18058 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:45.223171 17119 catalog_manager.cc:5484] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:45.223603 17791 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:45.225828 17398 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:45.228793 17529 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:45.228807 17924 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:45.228935 17397 consensus_queue.cc:237] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:45.230665 17791 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.230948 17925 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.231029 20302 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.231544 18057 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.231978 17266 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:45.232376 20300 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.232678 20300 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.233443 17660 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.234463 20299 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.234542 17266 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.236270 17529 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:45.236268 20302 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.236629 20299 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:45.236594 20302 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:45.237417 17660 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:45.237900 17529 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:45.238543 17791 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:45.239589 17124 catalog_manager.cc:5484] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:45.240128 17924 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:45.243902 17266 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:45.244021 18057 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:45.247558 17397 consensus_queue.cc:237] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:45.249482 18057 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:45.249709 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138:33637): Couldn't send request to peer ab18888a9724446b9ee4fb9efe811b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d1de048a83e45548afac96309abf4d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:45.249831 17924 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.249979 17266 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.249979 17660 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.250087 17529 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.250336 19964 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.250455 19907 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.250473 17791 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:45.250557 20300 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.250828 19964 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.251075 19892 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.251286 20300 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:45.253878 20300 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:45.254310 17924 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:45.254765 18057 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:45.257236 17124 catalog_manager.cc:5484] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:45.257166 17791 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:45.257686 17660 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:45.259637 17529 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:45.261655 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3d1de048a83e45548afac96309abf4d2 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:45.261986 17266 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:45.309026 20601 ts_tablet_manager.cc:902] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:45.309499 20601 tablet_copy_client.cc:308] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:45.312994 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet e57bf40798824baf8050988874664154 from peer ab18888a9724446b9ee4fb9efe811b09 ({username='slave'} at 127.0.99.138:43825)
I20240503 08:00:45.313128 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet e57bf40798824baf8050988874664154 from peer ab18888a9724446b9ee4fb9efe811b09 at {username='slave'} at 127.0.99.138:43825: session id = ab18888a9724446b9ee4fb9efe811b09-e57bf40798824baf8050988874664154
I20240503 08:00:45.314009 17946 tablet_copy_source_session.cc:205] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:45.314849 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:45.316898 20601 tablet_copy_client.cc:783] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:45.317031 20601 tablet_copy_client.cc:647] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:45.318429 20601 tablet_copy_client.cc:523] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:45.319460 20601 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap starting.
I20240503 08:00:45.335716 20601 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:45.336052 20601 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap complete.
I20240503 08:00:45.336200 20601 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20240503 08:00:45.336555 20601 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:45.336655 20601 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab18888a9724446b9ee4fb9efe811b09, State: Initialized, Role: LEARNER
I20240503 08:00:45.336745 20601 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:45.336992 20601 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:45.337466 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session ab18888a9724446b9ee4fb9efe811b09-e57bf40798824baf8050988874664154 received from {username='slave'} at 127.0.99.138:43825
I20240503 08:00:45.337551 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session ab18888a9724446b9ee4fb9efe811b09-e57bf40798824baf8050988874664154 on tablet e57bf40798824baf8050988874664154 with peer ab18888a9724446b9ee4fb9efe811b09
I20240503 08:00:45.373819 20528 ts_tablet_manager.cc:902] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:45.374235 20528 tablet_copy_client.cc:308] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39647
I20240503 08:00:45.377919 17550 tablet_copy_service.cc:140] P 7af8f2a481294a2cb93e5761947978cc: Received BeginTabletCopySession request for tablet 5154e7f64d1542979f9363a429fa469c from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:34313)
I20240503 08:00:45.378015 17550 tablet_copy_service.cc:161] P 7af8f2a481294a2cb93e5761947978cc: Beginning new tablet copy session on tablet 5154e7f64d1542979f9363a429fa469c from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:34313: session id = b65378a282b9443484843de0b8906df9-5154e7f64d1542979f9363a429fa469c
I20240503 08:00:45.378808 17550 tablet_copy_source_session.cc:205] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:45.379539 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154e7f64d1542979f9363a429fa469c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:45.381209 20528 tablet_copy_client.cc:783] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:45.381351 20528 tablet_copy_client.cc:647] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:45.382315 20528 tablet_copy_client.cc:523] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:45.383184 20528 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:45.401800 20528 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:45.402079 20528 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:45.402432 20528 ts_tablet_manager.cc:1367] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.019s user 0.020s sys 0.000s
I20240503 08:00:45.402832 20528 raft_consensus.cc:348] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:45.402940 20528 raft_consensus.cc:729] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:45.403028 20528 consensus_queue.cc:260] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:45.403270 20528 ts_tablet_manager.cc:1398] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:45.403892 17550 tablet_copy_service.cc:342] P 7af8f2a481294a2cb93e5761947978cc: Request end of tablet copy session b65378a282b9443484843de0b8906df9-5154e7f64d1542979f9363a429fa469c received from {username='slave'} at 127.0.99.136:34313
I20240503 08:00:45.403972 17550 tablet_copy_service.cc:433] P 7af8f2a481294a2cb93e5761947978cc: ending tablet copy session b65378a282b9443484843de0b8906df9-5154e7f64d1542979f9363a429fa469c on tablet 5154e7f64d1542979f9363a429fa469c with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:45.411711 20601 ts_tablet_manager.cc:902] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: Initiating tablet copy from peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825)
I20240503 08:00:45.412097 20601 tablet_copy_client.cc:308] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38825
I20240503 08:00:45.415189 17814 tablet_copy_service.cc:140] P 84305d92b082406380b107114e4cd61b: Received BeginTabletCopySession request for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 from peer ab18888a9724446b9ee4fb9efe811b09 ({username='slave'} at 127.0.99.138:48317)
I20240503 08:00:45.415294 17814 tablet_copy_service.cc:161] P 84305d92b082406380b107114e4cd61b: Beginning new tablet copy session on tablet bbe0edacc8ce4ba8be5e533c82ea92e0 from peer ab18888a9724446b9ee4fb9efe811b09 at {username='slave'} at 127.0.99.138:48317: session id = ab18888a9724446b9ee4fb9efe811b09-bbe0edacc8ce4ba8be5e533c82ea92e0
I20240503 08:00:45.416302 17814 tablet_copy_source_session.cc:205] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:45.417279 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe0edacc8ce4ba8be5e533c82ea92e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:45.418771 20601 tablet_copy_client.cc:783] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:45.418892 20601 tablet_copy_client.cc:647] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:45.420094 20601 tablet_copy_client.cc:523] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:45.420986 20601 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap starting.
I20240503 08:00:45.438081 20601 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:45.438282 20601 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap complete.
I20240503 08:00:45.438413 20601 ts_tablet_manager.cc:1367] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20240503 08:00:45.438748 20601 raft_consensus.cc:348] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:45.438849 20601 raft_consensus.cc:729] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab18888a9724446b9ee4fb9efe811b09, State: Initialized, Role: LEARNER
I20240503 08:00:45.438936 20601 consensus_queue.cc:260] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:45.439581 20601 ts_tablet_manager.cc:1398] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:45.440054 17814 tablet_copy_service.cc:342] P 84305d92b082406380b107114e4cd61b: Request end of tablet copy session ab18888a9724446b9ee4fb9efe811b09-bbe0edacc8ce4ba8be5e533c82ea92e0 received from {username='slave'} at 127.0.99.138:48317
I20240503 08:00:45.440135 17814 tablet_copy_service.cc:433] P 84305d92b082406380b107114e4cd61b: ending tablet copy session ab18888a9724446b9ee4fb9efe811b09-bbe0edacc8ce4ba8be5e533c82ea92e0 on tablet bbe0edacc8ce4ba8be5e533c82ea92e0 with peer ab18888a9724446b9ee4fb9efe811b09
I20240503 08:00:45.469470 20528 ts_tablet_manager.cc:902] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:45.469926 20528 tablet_copy_client.cc:308] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:41359
I20240503 08:00:45.473610 18078 tablet_copy_service.cc:140] P d502c5c7f12a4e39aa25cd57f92c403d: Received BeginTabletCopySession request for tablet 08e9b83cd82f4d82ae552501be742896 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:56169)
I20240503 08:00:45.473785 18078 tablet_copy_service.cc:161] P d502c5c7f12a4e39aa25cd57f92c403d: Beginning new tablet copy session on tablet 08e9b83cd82f4d82ae552501be742896 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:56169: session id = b65378a282b9443484843de0b8906df9-08e9b83cd82f4d82ae552501be742896
I20240503 08:00:45.474927 18078 tablet_copy_source_session.cc:205] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:45.476006 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e9b83cd82f4d82ae552501be742896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:45.477701 20528 tablet_copy_client.cc:783] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:45.477860 20528 tablet_copy_client.cc:647] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:45.479207 20528 tablet_copy_client.cc:523] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:45.480315 20528 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:45.501042 20528 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:45.501312 20528 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:45.501457 20528 ts_tablet_manager.cc:1367] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.005s
I20240503 08:00:45.501821 20528 raft_consensus.cc:348] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:45.501929 20528 raft_consensus.cc:729] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:45.502049 20528 consensus_queue.cc:260] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:45.502319 20528 ts_tablet_manager.cc:1398] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:45.502987 18078 tablet_copy_service.cc:342] P d502c5c7f12a4e39aa25cd57f92c403d: Request end of tablet copy session b65378a282b9443484843de0b8906df9-08e9b83cd82f4d82ae552501be742896 received from {username='slave'} at 127.0.99.136:56169
I20240503 08:00:45.503091 18078 tablet_copy_service.cc:433] P d502c5c7f12a4e39aa25cd57f92c403d: ending tablet copy session b65378a282b9443484843de0b8906df9-08e9b83cd82f4d82ae552501be742896 on tablet 08e9b83cd82f4d82ae552501be742896 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:45.675493 20528 ts_tablet_manager.cc:902] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863)
I20240503 08:00:45.675936 20528 tablet_copy_client.cc:308] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:34863
I20240503 08:00:45.676302 17682 tablet_copy_service.cc:140] P 0ce150b975ff4040bedde6319d5cf3d5: Received BeginTabletCopySession request for tablet ba0615c2fbc840c7bbdfda22caf45de2 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:49971)
I20240503 08:00:45.676407 17682 tablet_copy_service.cc:161] P 0ce150b975ff4040bedde6319d5cf3d5: Beginning new tablet copy session on tablet ba0615c2fbc840c7bbdfda22caf45de2 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:49971: session id = b65378a282b9443484843de0b8906df9-ba0615c2fbc840c7bbdfda22caf45de2
I20240503 08:00:45.677004 19636 raft_consensus.cc:1203] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:45.677361 17682 tablet_copy_source_session.cc:205] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:45.678277 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0615c2fbc840c7bbdfda22caf45de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:45.680254 20528 tablet_copy_client.cc:783] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:45.680392 20528 tablet_copy_client.cc:647] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:45.681449 20528 tablet_copy_client.cc:523] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:45.682515 20528 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:45.699081 20528 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:45.699348 20528 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:45.699522 20528 ts_tablet_manager.cc:1367] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.001s
I20240503 08:00:45.699968 20528 raft_consensus.cc:348] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:45.700106 20528 raft_consensus.cc:729] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:45.700209 20528 consensus_queue.cc:260] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:45.700469 20528 ts_tablet_manager.cc:1398] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:45.700902 17682 tablet_copy_service.cc:342] P 0ce150b975ff4040bedde6319d5cf3d5: Request end of tablet copy session b65378a282b9443484843de0b8906df9-ba0615c2fbc840c7bbdfda22caf45de2 received from {username='slave'} at 127.0.99.136:49971
I20240503 08:00:45.700975 17682 tablet_copy_service.cc:433] P 0ce150b975ff4040bedde6319d5cf3d5: ending tablet copy session b65378a282b9443484843de0b8906df9-ba0615c2fbc840c7bbdfda22caf45de2 on tablet ba0615c2fbc840c7bbdfda22caf45de2 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:45.717800 19505 raft_consensus.cc:1203] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:45.745218 20601 ts_tablet_manager.cc:902] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:45.745646 20601 tablet_copy_client.cc:308] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:45.746068 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet 3d1de048a83e45548afac96309abf4d2 from peer ab18888a9724446b9ee4fb9efe811b09 ({username='slave'} at 127.0.99.138:48513)
I20240503 08:00:45.746169 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet 3d1de048a83e45548afac96309abf4d2 from peer ab18888a9724446b9ee4fb9efe811b09 at {username='slave'} at 127.0.99.138:48513: session id = ab18888a9724446b9ee4fb9efe811b09-3d1de048a83e45548afac96309abf4d2
I20240503 08:00:45.746896 17418 tablet_copy_source_session.cc:205] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:45.747766 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1de048a83e45548afac96309abf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:45.749724 20601 tablet_copy_client.cc:783] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:45.749855 20601 tablet_copy_client.cc:647] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:45.750929 20601 tablet_copy_client.cc:523] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:45.752135 20601 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap starting.
I20240503 08:00:45.773559 20601 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:45.774021 20601 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap complete.
I20240503 08:00:45.774199 20601 ts_tablet_manager.cc:1367] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240503 08:00:45.774593 20601 raft_consensus.cc:348] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:45.774705 20601 raft_consensus.cc:729] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab18888a9724446b9ee4fb9efe811b09, State: Initialized, Role: LEARNER
I20240503 08:00:45.774803 20601 consensus_queue.cc:260] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:45.775074 20601 ts_tablet_manager.cc:1398] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:45.775537 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session ab18888a9724446b9ee4fb9efe811b09-3d1de048a83e45548afac96309abf4d2 received from {username='slave'} at 127.0.99.138:48513
I20240503 08:00:45.775621 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session ab18888a9724446b9ee4fb9efe811b09-3d1de048a83e45548afac96309abf4d2 on tablet 3d1de048a83e45548afac96309abf4d2 with peer ab18888a9724446b9ee4fb9efe811b09
I20240503 08:00:45.888852 19636 raft_consensus.cc:1203] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:45.979539 19505 raft_consensus.cc:1203] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:46.004173 19636 raft_consensus.cc:1203] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:46.118230 19636 raft_consensus.cc:1203] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:46.168617 19505 raft_consensus.cc:1203] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:46.186399 20328 raft_consensus.cc:1050] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:46.186869 20328 consensus_queue.cc:237] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:46.188589 17791 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.188905 17265 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.189172 18057 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.189393 17397 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.189359 20750 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.189491 19636 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.189828 17924 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.189831 20749 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.190397 20749 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.190800 20325 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.190896 17530 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.191088 20325 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.192371 20748 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.194130 20750 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.194522 20325 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.196743 19636 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.198338 20665 raft_consensus.cc:1050] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER ab18888a9724446b9ee4fb9efe811b09 to VOTER
I20240503 08:00:46.198537 17120 catalog_manager.cc:5484] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:46.198860 20665 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:46.200294 17791 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.201344 17793 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.201514 19505 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.201597 17660 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.201758 18058 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.201975 20663 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.202564 20316 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.202701 20663 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.203004 20663 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.204810 17265 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.204968 17397 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.205343 20761 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.205562 17530 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.206077 20316 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.206259 20764 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.206727 17398 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.208163 17529 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.210166 17924 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.210507 17265 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.211670 18057 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.215853 17660 consensus_queue.cc:237] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:46.216694 20663 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.217764 18057 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.218005 17793 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.218117 20750 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.218376 17397 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.219043 17793 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.219471 20748 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.219732 19636 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.220139 20748 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.220254 17397 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.220832 17124 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:46.220934 17266 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.221324 20748 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.222129 17265 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.222429 17529 raft_consensus.cc:1261] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.222966 17660 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.223771 20750 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.224087 20750 consensus_queue.cc:1035] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.225024 20749 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.225368 19636 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.226410 17791 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.229133 17117 catalog_manager.cc:5484] T ecc1f8b69268429f89cffb380e45e020 P b65378a282b9443484843de0b8906df9 reported cstate change: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.229153 17265 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.231288 17905 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc1f8b69268429f89cffb380e45e020 with delete_type TABLET_DATA_TOMBSTONED (TS 78e52cf3c8fa435bb5ca36061227ac41 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34568
I20240503 08:00:46.231566 17529 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.232113 17397 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.233023 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ecc1f8b69268429f89cffb380e45e020 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:46.233871 18058 raft_consensus.cc:2932] T ecc1f8b69268429f89cffb380e45e020 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.234676 19505 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.237490 17530 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.238373 18058 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.241331 20778 tablet_replica.cc:330] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41: stopping tablet replica
I20240503 08:00:46.241472 20778 raft_consensus.cc:2229] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:46.241637 20778 raft_consensus.cc:2258] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:46.242322 20778 ts_tablet_manager.cc:1875] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:46.244072 17924 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:46.244939 20778 ts_tablet_manager.cc:1888] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:46.245044 20778 log.cc:1196] T ecc1f8b69268429f89cffb380e45e020 P 78e52cf3c8fa435bb5ca36061227ac41: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ecc1f8b69268429f89cffb380e45e020
I20240503 08:00:46.245950 19505 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.246101 17265 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.246189 17660 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.246424 17109 catalog_manager.cc:4830] TS 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175): tablet ecc1f8b69268429f89cffb380e45e020 (table rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]) successfully deleted
I20240503 08:00:46.246479 17530 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.246598 20665 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.246809 20760 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.246937 20764 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.247272 18058 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.247257 20663 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.247601 20663 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.249325 17397 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.249622 20760 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.250378 20760 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.251013 19505 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.251094 17660 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.252471 17109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e57bf40798824baf8050988874664154 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:46.252831 18058 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.253707 17117 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.255595 17265 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.256019 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet e57bf40798824baf8050988874664154 with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:46.256778 17530 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.257289 17397 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.257830 20784 tablet_replica.cc:330] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:46.257951 20784 raft_consensus.cc:2229] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:46.258394 20784 raft_consensus.cc:2258] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:46.260282 20784 ts_tablet_manager.cc:1875] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:46.262722 20784 ts_tablet_manager.cc:1888] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:46.262799 20784 log.cc:1196] T e57bf40798824baf8050988874664154 P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e57bf40798824baf8050988874664154
I20240503 08:00:46.263259 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet e57bf40798824baf8050988874664154 (table rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]) successfully deleted
I20240503 08:00:46.410815 20708 raft_consensus.cc:1050] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b: attempting to promote NON_VOTER ab18888a9724446b9ee4fb9efe811b09 to VOTER
I20240503 08:00:46.411350 20708 consensus_queue.cc:237] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:46.412880 19505 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.413236 17265 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.413479 17397 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.413496 17924 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.413628 20813 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.413841 17660 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.413849 20814 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.414113 20815 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.414175 18057 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.414731 20814 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.415172 17530 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.415180 20815 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.415529 20472 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.417032 20815 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.418123 20472 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.418437 19505 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.418957 17924 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.419049 18057 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.421414 17660 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.421648 17530 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.422180 17119 catalog_manager.cc:5484] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84305d92b082406380b107114e4cd61b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.423086 17265 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.424517 17397 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.429622 17791 consensus_queue.cc:237] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:46.430866 19505 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.431164 17265 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.431214 17924 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.431450 17397 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.431490 18057 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.431568 20813 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.431708 20815 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.431854 20814 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.432003 20816 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.432180 17530 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.432179 20812 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.432543 20815 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.434155 20813 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.434651 17530 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.435739 17924 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.436007 17110 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:46.435941 17397 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.436187 19505 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.436515 17265 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.438382 18057 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.438511 17123 catalog_manager.cc:5484] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b reported cstate change: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "84305d92b082406380b107114e4cd61b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:46.441135 17640 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 with delete_type TABLET_DATA_TOMBSTONED (TS 0ce150b975ff4040bedde6319d5cf3d5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35622
I20240503 08:00:46.442885 20824 tablet_replica.cc:330] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5: stopping tablet replica
I20240503 08:00:46.443056 20824 raft_consensus.cc:2229] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:46.443271 20824 raft_consensus.cc:2258] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:46.444490 20824 ts_tablet_manager.cc:1875] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:46.446210 20824 ts_tablet_manager.cc:1888] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:46.446269 20824 log.cc:1196] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 0ce150b975ff4040bedde6319d5cf3d5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/bbe0edacc8ce4ba8be5e533c82ea92e0
I20240503 08:00:46.446691 17108 catalog_manager.cc:4830] TS 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863): tablet bbe0edacc8ce4ba8be5e533c82ea92e0 (table rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]) successfully deleted
I20240503 08:00:46.448239 20707 raft_consensus.cc:1050] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:46.448645 20707 consensus_queue.cc:237] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:46.450385 17924 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.450559 17791 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.450603 17265 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.450726 20826 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.450963 20243 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.451078 18057 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.451196 19636 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.451581 17397 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.451577 20243 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.451854 20243 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.452081 17660 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.452350 20826 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.452497 20828 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.453723 20827 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.454329 20707 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.454746 17397 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.454960 17660 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.456240 19636 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.457799 17118 catalog_manager.cc:5484] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.458019 18057 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.460107 17924 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.460304 17791 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.463438 17265 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.467839 17530 consensus_queue.cc:237] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:46.469439 17791 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.469444 17924 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.469679 18057 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.469780 20827 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.470007 17397 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.470007 20826 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.470088 17265 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.470206 20243 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.470448 20826 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.470718 20243 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.471354 19636 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.471712 20707 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.472492 20826 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.473115 17791 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.473695 19635 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.473937 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5154e7f64d1542979f9363a429fa469c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:46.474548 17265 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.474588 17924 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.475557 20441 raft_consensus.cc:1050] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:46.476087 20441 consensus_queue.cc:237] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:46.476976 18057 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.477160 17117 catalog_manager.cc:5484] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.479296 17397 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.479542 17265 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.479604 19635 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.480511 17660 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.480149 20843 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.480602 17398 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.481009 17530 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.481364 17924 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.480829 20842 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 08:00:46.482260 20843 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.482605 20843 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.483280 17640 tablet_service.cc:1514] Processing DeleteTablet for tablet 5154e7f64d1542979f9363a429fa469c with delete_type TABLET_DATA_TOMBSTONED (TS 0ce150b975ff4040bedde6319d5cf3d5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35622
I20240503 08:00:46.483800 20824 tablet_replica.cc:330] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5: stopping tablet replica
I20240503 08:00:46.483940 20824 raft_consensus.cc:2229] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:46.484143 20824 raft_consensus.cc:2258] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:46.485105 20824 ts_tablet_manager.cc:1875] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:46.486001 17791 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.487165 20824 ts_tablet_manager.cc:1888] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:46.487224 20824 log.cc:1196] T 5154e7f64d1542979f9363a429fa469c P 0ce150b975ff4040bedde6319d5cf3d5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5154e7f64d1542979f9363a429fa469c
I20240503 08:00:46.487619 17108 catalog_manager.cc:4830] TS 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863): tablet 5154e7f64d1542979f9363a429fa469c (table rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]) successfully deleted
I20240503 08:00:46.482987 20840 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20240503 08:00:46.489426 20839 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.489686 20441 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.491523 20838 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.493321 17265 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.493484 17398 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.493484 17924 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.495647 17118 catalog_manager.cc:5484] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.496182 17530 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.499330 17791 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.501140 19635 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.503674 17660 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.506085 18057 consensus_queue.cc:237] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:46.507802 17398 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.507897 17530 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.507939 17265 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.508244 20842 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.508311 17660 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.508343 19635 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.508419 20840 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.508538 20838 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.508697 20843 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.508931 20839 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.511730 20840 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.512064 20326 raft_consensus.cc:1050] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:46.512269 17924 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.512167 17265 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.512537 20326 consensus_queue.cc:237] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:46.512593 20842 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.513572 19635 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.513586 17924 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.514705 17791 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.514925 19636 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.515086 20748 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.515724 17397 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.515724 17530 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.516073 20748 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.516296 20325 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.516568 20325 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.517956 17398 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.518118 18057 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.518100 17530 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.518450 20325 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.518467 17109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08e9b83cd82f4d82ae552501be742896 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:46.519122 17120 catalog_manager.cc:5484] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.519870 17925 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.520251 20325 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.522619 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e9b83cd82f4d82ae552501be742896 with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:46.522974 20784 tablet_replica.cc:330] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:46.523144 20784 raft_consensus.cc:2229] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:46.523336 20784 raft_consensus.cc:2258] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:46.523931 17660 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.524056 20784 ts_tablet_manager.cc:1875] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:46.525907 17265 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.526194 20784 ts_tablet_manager.cc:1888] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:46.526254 20784 log.cc:1196] T 08e9b83cd82f4d82ae552501be742896 P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/08e9b83cd82f4d82ae552501be742896
I20240503 08:00:46.526275 20325 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.526816 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet 08e9b83cd82f4d82ae552501be742896 (table rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]) successfully deleted
I20240503 08:00:46.527729 20325 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.529506 17924 consensus_queue.cc:237] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:46.530113 19636 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.530151 17397 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.530395 18057 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.531509 17791 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.532639 20761 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.532713 17530 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.532830 17266 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.532716 17124 catalog_manager.cc:5484] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.533299 18057 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.533527 17397 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.533689 20764 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.534027 17660 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.534025 17924 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.534642 20316 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.534981 20316 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.535974 20764 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.535843 17530 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.536254 20764 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.539801 17793 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.540014 20764 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.540432 17660 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.541098 17266 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.542312 18057 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.543619 17266 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.545532 17397 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.546690 17124 catalog_manager.cc:5484] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.547632 17660 consensus_queue.cc:237] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:46.549268 17529 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.549294 18057 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.549608 20326 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.549788 20750 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.549916 19636 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.550796 17397 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.551019 20750 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.551189 17266 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.551554 20326 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.551859 20326 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.552168 17924 consensus_queue.cc:237] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:46.552352 17926 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.552740 20750 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.553725 20326 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.554435 18057 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.554512 17397 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.554935 20663 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.555650 17266 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.555917 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba0615c2fbc840c7bbdfda22caf45de2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:46.556066 17397 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.556519 17528 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.557626 19636 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.558193 20316 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.558439 20316 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.558339 17265 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.559240 17660 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.559845 17924 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:46.557662 17120 catalog_manager.cc:5484] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:46.562167 20761 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.562847 18057 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
W20240503 08:00:46.567227 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15a5f02e61d24029b0a5757556085733. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:46.567378 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet ba0615c2fbc840c7bbdfda22caf45de2 with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:46.567667 20784 tablet_replica.cc:330] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:46.568390 20784 raft_consensus.cc:2229] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:46.568504 17529 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.568522 17793 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.568562 20784 raft_consensus.cc:2258] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:46.569602 20784 ts_tablet_manager.cc:1875] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:46.571347 20784 ts_tablet_manager.cc:1888] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:46.571406 20784 log.cc:1196] T ba0615c2fbc840c7bbdfda22caf45de2 P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ba0615c2fbc840c7bbdfda22caf45de2
I20240503 08:00:46.571763 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet ba0615c2fbc840c7bbdfda22caf45de2 (table rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]) successfully deleted
I20240503 08:00:46.572149 20761 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.572842 17793 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.573076 20764 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.573282 17397 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:46.573348 20316 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:46.575026 17529 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.575299 17793 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:46.575690 17660 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:46.576421 17529 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:46.577129 17109 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 15a5f02e61d24029b0a5757556085733 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:46.577455 17124 catalog_manager.cc:5484] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:46.578770 18057 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:46.581514 17266 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:46.762670 20300 raft_consensus.cc:1050] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298: attempting to promote NON_VOTER ab18888a9724446b9ee4fb9efe811b09 to VOTER
I20240503 08:00:46.763147 20300 consensus_queue.cc:237] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:46.764827 17660 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.764844 17924 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.764899 19505 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.765116 17266 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.765378 20906 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.765570 20908 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.765600 18057 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.765875 20906 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.766016 17529 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.766180 20906 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.766536 20906 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.767016 17793 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:46.767385 19892 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.768024 20907 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:46.769125 19892 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.769913 17924 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.770148 17529 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.771292 17660 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.771621 19505 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.772050 17793 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.772675 17121 catalog_manager.cc:5484] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:46.772724 17266 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.774536 18057 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.777442 17397 consensus_queue.cc:237] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:46.779165 18057 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.779809 17529 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.779855 17265 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.780148 20300 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.780324 20908 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.780464 20906 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.780651 19505 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.780987 17793 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.781024 20905 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.781369 20905 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.781394 17924 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:46.785324 20905 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:46.787241 20906 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.787643 19505 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.787886 17793 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.789427 18057 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.789796 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d1de048a83e45548afac96309abf4d2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:46.790155 17265 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.790237 17124 catalog_manager.cc:5484] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:46.791368 17529 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.792783 17924 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:46.794049 17640 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1de048a83e45548afac96309abf4d2 with delete_type TABLET_DATA_TOMBSTONED (TS 0ce150b975ff4040bedde6319d5cf3d5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35622
I20240503 08:00:46.794286 20824 tablet_replica.cc:330] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5: stopping tablet replica
I20240503 08:00:46.794432 20824 raft_consensus.cc:2229] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:46.794611 20824 raft_consensus.cc:2258] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:46.796506 20824 ts_tablet_manager.cc:1875] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:46.798388 20824 ts_tablet_manager.cc:1888] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:46.798449 20824 log.cc:1196] T 3d1de048a83e45548afac96309abf4d2 P 0ce150b975ff4040bedde6319d5cf3d5: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d1de048a83e45548afac96309abf4d2
I20240503 08:00:46.798843 17108 catalog_manager.cc:4830] TS 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863): tablet 3d1de048a83e45548afac96309abf4d2 (table rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]) successfully deleted
I20240503 08:00:46.810396 17397 consensus_queue.cc:237] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:46.812081 17924 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.812471 20905 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.812604 17660 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.812724 18057 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.813041 17793 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.813712 20300 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.813745 17529 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.813903 17265 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.814059 20907 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.814314 20300 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.814585 20907 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.815059 20300 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.816294 19892 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.816807 18057 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.819507 17529 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.819876 17123 catalog_manager.cc:5484] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.820209 17924 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.820922 17266 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.822775 17660 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.825685 17793 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.828446 17397 consensus_queue.cc:237] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:46.830245 18057 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.830245 17924 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:46.830374 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 304ad5f1abf44548a72961d3f655cf61. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:46.830590 20300 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.830754 17793 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.830758 20906 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.831156 20300 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.832403 17529 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.832420 17660 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.832872 20908 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.832912 17266 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.833319 20905 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.833477 20907 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.833612 20906 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.833976 17529 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.834401 17660 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.836371 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 304ad5f1abf44548a72961d3f655cf61 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:46.836218 17924 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.836568 18057 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.836694 17117 catalog_manager.cc:5484] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.837038 17266 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.833359 17793 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.843277 17266 consensus_queue.cc:237] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } }
I20240503 08:00:46.845290 17924 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.845351 18057 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.845606 17660 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.845896 17793 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.845860 20785 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.846189 20785 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.846482 17397 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.846483 17529 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:46.846478 20785 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.846735 20785 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.847011 20785 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.847283 20785 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:46.850286 20938 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.850764 17660 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.850791 17397 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.852250 17924 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.852867 17529 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.853309 17120 catalog_manager.cc:5484] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:46.854626 17793 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.856654 18058 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } }
I20240503 08:00:46.860476 17266 consensus_queue.cc:237] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:46.862396 17793 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.863073 18057 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.863310 17660 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.863327 17529 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.863662 17397 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.863866 20939 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.864203 20939 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.864588 20939 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.865533 17924 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:46.866263 20938 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.866470 20928 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.866591 20938 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:46.868294 20940 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.869486 17660 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.870033 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 00708b0b8f2f4366944b244a9ba8c29f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:46.870451 17397 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.870463 17791 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.870575 17924 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.872583 17529 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.872789 17117 catalog_manager.cc:5484] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:46.874843 18057 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
W20240503 08:00:46.881814 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00708b0b8f2f4366944b244a9ba8c29f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:46.979555 20969 ts_tablet_manager.cc:902] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:46.980024 20969 tablet_copy_client.cc:308] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:46.983300 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet 15a5f02e61d24029b0a5757556085733 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:37051)
I20240503 08:00:46.983412 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet 15a5f02e61d24029b0a5757556085733 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:37051: session id = b65378a282b9443484843de0b8906df9-15a5f02e61d24029b0a5757556085733
I20240503 08:00:46.984099 17946 tablet_copy_source_session.cc:205] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:46.984915 20969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a5f02e61d24029b0a5757556085733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:46.986791 20969 tablet_copy_client.cc:783] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:46.986932 20969 tablet_copy_client.cc:647] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:46.987952 20969 tablet_copy_client.cc:523] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:46.988894 20969 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:47.007506 20969 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:47.007761 20969 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:47.007936 20969 ts_tablet_manager.cc:1367] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240503 08:00:47.008289 20969 raft_consensus.cc:348] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:47.008399 20969 raft_consensus.cc:729] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:47.008498 20969 consensus_queue.cc:260] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:47.008749 20969 ts_tablet_manager.cc:1398] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:47.017164 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session b65378a282b9443484843de0b8906df9-15a5f02e61d24029b0a5757556085733 received from {username='slave'} at 127.0.99.136:37051
I20240503 08:00:47.017254 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session b65378a282b9443484843de0b8906df9-15a5f02e61d24029b0a5757556085733 on tablet 15a5f02e61d24029b0a5757556085733 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:47.306248 20994 ts_tablet_manager.cc:902] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:47.306805 20994 tablet_copy_client.cc:308] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:47.307286 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet 304ad5f1abf44548a72961d3f655cf61 from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:50179)
I20240503 08:00:47.307394 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet 304ad5f1abf44548a72961d3f655cf61 from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:50179: session id = 0665f4f85b984a8bba213ad479a88bad-304ad5f1abf44548a72961d3f655cf61
I20240503 08:00:47.308223 17418 tablet_copy_source_session.cc:205] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:47.309170 20994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304ad5f1abf44548a72961d3f655cf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:47.311333 20994 tablet_copy_client.cc:783] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:47.311486 20994 tablet_copy_client.cc:647] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:47.312692 20994 tablet_copy_client.cc:523] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:47.313889 20994 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:47.333791 20994 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:47.334014 20994 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:47.334165 20994 ts_tablet_manager.cc:1367] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.002s
I20240503 08:00:47.334519 20994 raft_consensus.cc:348] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:47.334625 20994 raft_consensus.cc:729] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:47.334715 20994 consensus_queue.cc:260] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:47.335023 20994 ts_tablet_manager.cc:1398] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:47.335461 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-304ad5f1abf44548a72961d3f655cf61 received from {username='slave'} at 127.0.99.137:50179
I20240503 08:00:47.335542 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-304ad5f1abf44548a72961d3f655cf61 on tablet 304ad5f1abf44548a72961d3f655cf61 with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:47.417922 19636 raft_consensus.cc:1203] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:47.443464 20994 ts_tablet_manager.cc:902] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:47.443898 20994 tablet_copy_client.cc:308] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33199
I20240503 08:00:47.447180 17286 tablet_copy_service.cc:140] P 8eb6822290b046f88c861159aa9421ec: Received BeginTabletCopySession request for tablet 00708b0b8f2f4366944b244a9ba8c29f from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:44925)
I20240503 08:00:47.447273 17286 tablet_copy_service.cc:161] P 8eb6822290b046f88c861159aa9421ec: Beginning new tablet copy session on tablet 00708b0b8f2f4366944b244a9ba8c29f from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:44925: session id = 0665f4f85b984a8bba213ad479a88bad-00708b0b8f2f4366944b244a9ba8c29f
I20240503 08:00:47.447937 17286 tablet_copy_source_session.cc:205] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:47.448732 20994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00708b0b8f2f4366944b244a9ba8c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:47.450441 20994 tablet_copy_client.cc:783] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:47.450603 20994 tablet_copy_client.cc:647] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:47.451680 20994 tablet_copy_client.cc:523] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:47.452759 20994 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:47.471334 20994 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:47.471560 20994 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:47.471715 20994 ts_tablet_manager.cc:1367] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20240503 08:00:47.472126 20994 raft_consensus.cc:348] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:47.472236 20994 raft_consensus.cc:729] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:47.472326 20994 consensus_queue.cc:260] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:47.472590 20994 ts_tablet_manager.cc:1398] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240503 08:00:47.473500 17286 tablet_copy_service.cc:342] P 8eb6822290b046f88c861159aa9421ec: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-00708b0b8f2f4366944b244a9ba8c29f received from {username='slave'} at 127.0.99.137:44925
I20240503 08:00:47.473593 17286 tablet_copy_service.cc:433] P 8eb6822290b046f88c861159aa9421ec: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-00708b0b8f2f4366944b244a9ba8c29f on tablet 00708b0b8f2f4366944b244a9ba8c29f with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:47.760694 19374 raft_consensus.cc:1203] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:47.796734 21017 raft_consensus.cc:1050] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:47.797286 21017 consensus_queue.cc:237] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:47.798975 19636 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:47.799369 18057 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:47.799500 17266 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:47.799521 17791 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:47.799750 17530 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:47.799724 21056 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:47.799825 17660 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:47.800145 21057 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:47.800256 17397 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:47.800310 21056 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:47.800441 20760 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:47.801229 21017 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:47.801403 21059 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:47.802625 21017 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:47.803910 21059 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.804652 17660 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.804826 17397 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.806025 17791 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.806533 17266 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.807438 18057 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.808115 17123 catalog_manager.cc:5484] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:47.808503 19636 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.812103 17530 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.814563 17926 consensus_queue.cc:237] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:47.816056 19636 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:47.816502 17397 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:47.816623 17660 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:47.816637 21058 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:47.816905 17266 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:47.817039 21058 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:47.817525 21058 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:47.817687 21056 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:47.817850 18057 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:47.818296 21058 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:47.818389 17530 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:47.820211 21057 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:47.820605 20993 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.821012 18057 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.823474 17109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 15a5f02e61d24029b0a5757556085733 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:47.823570 17124 catalog_manager.cc:5484] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:47.824004 17397 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.824261 17266 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.825495 17660 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.826686 17530 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.826797 19636 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:47.829027 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a5f02e61d24029b0a5757556085733 with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:47.830715 21070 tablet_replica.cc:330] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:47.830842 21070 raft_consensus.cc:2229] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:47.831018 21070 raft_consensus.cc:2258] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:47.831607 21070 ts_tablet_manager.cc:1875] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:47.833236 21070 ts_tablet_manager.cc:1888] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:47.833297 21070 log.cc:1196] T 15a5f02e61d24029b0a5757556085733 P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/15a5f02e61d24029b0a5757556085733
I20240503 08:00:47.833762 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet 15a5f02e61d24029b0a5757556085733 (table rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]) successfully deleted
I20240503 08:00:47.910310 19374 raft_consensus.cc:1203] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:48.179245 17926 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:48.180780 17397 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.180902 17530 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.181257 17266 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.181273 17791 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.181608 17660 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.182009 21056 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.182170 20993 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.182394 18057 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.182461 21017 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.182888 21058 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.183188 21058 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.184043 21055 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.185895 21017 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.186389 17530 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.186481 17266 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.187980 17791 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.188153 17660 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.189322 18057 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.190604 17115 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.197680 17397 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.199894 17926 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
W20240503 08:00:48.201714 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138:33637): Couldn't send request to peer ab18888a9724446b9ee4fb9efe811b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab988cc96fc940239e18f54ec9c9b401. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:48.202502 17660 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.202734 17266 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.202919 18057 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.203037 17791 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.203383 21055 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.203545 21057 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.203719 21056 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.203912 17397 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.204141 21059 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.204901 21056 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.206301 17530 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.206708 21057 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:48.207150 21059 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.207343 17660 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:48.208563 17109 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ab988cc96fc940239e18f54ec9c9b401 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:48.209329 17117 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:48.209344 17791 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:48.209806 17530 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:48.209918 17266 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:48.211418 17397 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:48.211977 18057 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } } }
I20240503 08:00:48.295107 21072 raft_consensus.cc:1050] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:48.295696 21072 consensus_queue.cc:237] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:48.297549 19374 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.297703 17397 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.297716 17926 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.297909 17660 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.298214 17791 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.298247 21119 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.298447 21120 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.298547 17530 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.298727 21072 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.298864 21119 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.299142 18057 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.299192 21118 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.299553 21072 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.300832 21120 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.302311 21119 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.303969 19374 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.304522 17926 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.304647 17660 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.305563 18057 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.305935 17115 catalog_manager.cc:5484] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:48.307132 17397 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.308187 17530 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.309788 17791 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.315026 17266 consensus_queue.cc:237] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:48.316458 18057 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.316576 17926 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.316874 21120 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.317035 20928 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.317231 17660 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.317267 17530 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.317679 19374 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.317988 17397 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.318293 20928 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.318718 21120 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.318845 21119 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.318989 20928 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.319849 20928 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.320184 19374 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.321946 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00708b0b8f2f4366944b244a9ba8c29f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:48.322455 17117 catalog_manager.cc:5484] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.322870 18057 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.323755 17661 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.323792 17926 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.325155 17398 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.326332 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet 00708b0b8f2f4366944b244a9ba8c29f with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:48.326521 21070 tablet_replica.cc:330] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:48.326638 21070 raft_consensus.cc:2229] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:48.326807 21070 raft_consensus.cc:2258] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:48.328538 21070 ts_tablet_manager.cc:1875] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:48.330430 21070 ts_tablet_manager.cc:1888] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:48.330502 21070 log.cc:1196] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/00708b0b8f2f4366944b244a9ba8c29f
I20240503 08:00:48.330603 17530 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.330940 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet 00708b0b8f2f4366944b244a9ba8c29f (table rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]) successfully deleted
I20240503 08:00:48.374195 20935 raft_consensus.cc:1050] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:48.374693 20935 consensus_queue.cc:237] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:48.376245 19374 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.376513 17924 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.376659 17266 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.376763 17530 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.377048 18057 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.377137 21016 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.377482 21016 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.377626 20935 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.377815 21133 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.377898 17661 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.378616 17791 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:48.378832 21132 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.379582 21129 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.381178 20935 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:48.382725 21130 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.383179 19374 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.383585 17661 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.385928 18057 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.386137 17119 catalog_manager.cc:5484] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad reported cstate change: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.387473 17924 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.388739 17791 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.389758 17266 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.394590 17530 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.397549 17397 consensus_queue.cc:237] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:48.399122 17924 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.399135 19374 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.399507 20935 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.399842 17530 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.399880 17661 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.400022 21131 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.400337 20935 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.400491 18057 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.400593 20935 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.400837 21016 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.401996 17266 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:48.402402 21131 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:48.403333 21130 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.403329 19374 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.403455 17661 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.404734 18057 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.404908 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 304ad5f1abf44548a72961d3f655cf61 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:48.405254 17924 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.406328 17266 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.407265 17117 catalog_manager.cc:5484] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.407475 17530 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:48.411163 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet 304ad5f1abf44548a72961d3f655cf61 with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:48.411422 21070 tablet_replica.cc:330] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:48.411548 21070 raft_consensus.cc:2229] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:48.411734 21070 raft_consensus.cc:2258] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:48.414144 21070 ts_tablet_manager.cc:1875] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:48.416035 21070 ts_tablet_manager.cc:1888] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:48.416090 21070 log.cc:1196] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/304ad5f1abf44548a72961d3f655cf61
I20240503 08:00:48.416488 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet 304ad5f1abf44548a72961d3f655cf61 (table rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]) successfully deleted
I20240503 08:00:48.614594 21156 ts_tablet_manager.cc:902] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:48.615900 21156 tablet_copy_client.cc:308] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:48.617800 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet ab988cc96fc940239e18f54ec9c9b401 from peer ab18888a9724446b9ee4fb9efe811b09 ({username='slave'} at 127.0.99.138:43825)
I20240503 08:00:48.618269 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet ab988cc96fc940239e18f54ec9c9b401 from peer ab18888a9724446b9ee4fb9efe811b09 at {username='slave'} at 127.0.99.138:43825: session id = ab18888a9724446b9ee4fb9efe811b09-ab988cc96fc940239e18f54ec9c9b401
I20240503 08:00:48.620292 17946 tablet_copy_source_session.cc:205] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:48.621361 21156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:48.623039 21156 tablet_copy_client.cc:783] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:48.623191 21156 tablet_copy_client.cc:647] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:48.624522 21156 tablet_copy_client.cc:523] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:48.625486 21156 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap starting.
I20240503 08:00:48.649384 21156 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:48.649695 21156 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: Bootstrap complete.
I20240503 08:00:48.649854 21156 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.003s
I20240503 08:00:48.650238 21156 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:48.650367 21156 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab18888a9724446b9ee4fb9efe811b09, State: Initialized, Role: LEARNER
I20240503 08:00:48.650446 21156 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: NON_VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: true } }
I20240503 08:00:48.650823 21156 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:48.651384 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session ab18888a9724446b9ee4fb9efe811b09-ab988cc96fc940239e18f54ec9c9b401 received from {username='slave'} at 127.0.99.138:43825
I20240503 08:00:48.651463 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session ab18888a9724446b9ee4fb9efe811b09-ab988cc96fc940239e18f54ec9c9b401 on tablet ab988cc96fc940239e18f54ec9c9b401 with peer ab18888a9724446b9ee4fb9efe811b09
I20240503 08:00:48.655633 17530 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } }
I20240503 08:00:48.657594 17661 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.657596 18057 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.657686 17791 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.657706 17266 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.658006 20707 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.658042 17924 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.658190 21171 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.658455 21172 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.658614 17397 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:00:48.658746 21171 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.660362 21172 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.660722 20707 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:00:48.662272 21171 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.662708 17397 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.663199 18057 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.664139 17924 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.664499 17791 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.666225 17118 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.669286 17266 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.672494 17530 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:48.673296 17661 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } }
I20240503 08:00:48.674185 17791 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:00:48.674589 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10e598a3a124472eab35c97689057ace. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:48.674820 18057 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.674854 21184 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.675321 21101 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.675431 17924 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.675886 17397 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.676231 21172 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.676911 17661 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.677291 21172 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.678992 17266 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:00:48.679385 21185 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:00:48.680938 21171 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:00:48.681355 21171 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:48.682300 18057 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:48.683168 17661 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:48.684736 17120 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:48.685436 17108 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 10e598a3a124472eab35c97689057ace with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:00:48.685835 17266 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:48.685835 17791 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:48.686290 17398 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:48.690035 17924 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:49.180352 21213 ts_tablet_manager.cc:902] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:49.180809 21213 tablet_copy_client.cc:308] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39647
I20240503 08:00:49.181249 17550 tablet_copy_service.cc:140] P 7af8f2a481294a2cb93e5761947978cc: Received BeginTabletCopySession request for tablet 10e598a3a124472eab35c97689057ace from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:34313)
I20240503 08:00:49.181427 17550 tablet_copy_service.cc:161] P 7af8f2a481294a2cb93e5761947978cc: Beginning new tablet copy session on tablet 10e598a3a124472eab35c97689057ace from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:34313: session id = b65378a282b9443484843de0b8906df9-10e598a3a124472eab35c97689057ace
I20240503 08:00:49.182432 17550 tablet_copy_source_session.cc:205] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:49.183423 21213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:49.185051 21213 tablet_copy_client.cc:783] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:49.185225 21213 tablet_copy_client.cc:647] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:49.186336 21213 tablet_copy_client.cc:523] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:49.187382 21213 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:49.204985 21213 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:49.205246 21213 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:49.205406 21213 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240503 08:00:49.205781 21213 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:49.205921 21213 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:49.206025 21213 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:49.206388 21213 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:49.206843 17550 tablet_copy_service.cc:342] P 7af8f2a481294a2cb93e5761947978cc: Request end of tablet copy session b65378a282b9443484843de0b8906df9-10e598a3a124472eab35c97689057ace received from {username='slave'} at 127.0.99.136:34313
I20240503 08:00:49.206924 17550 tablet_copy_service.cc:433] P 7af8f2a481294a2cb93e5761947978cc: ending tablet copy session b65378a282b9443484843de0b8906df9-10e598a3a124472eab35c97689057ace on tablet 10e598a3a124472eab35c97689057ace with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:49.221786 19505 raft_consensus.cc:1203] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:49.735489 19636 raft_consensus.cc:1203] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:00:49.738931 21221 raft_consensus.cc:1050] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER ab18888a9724446b9ee4fb9efe811b09 to VOTER
I20240503 08:00:49.739462 21221 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:49.741214 17398 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:49.741300 19505 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:49.741634 21220 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:49.741971 21220 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:49.742934 17791 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:49.743347 17266 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:49.743448 18057 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:49.743798 21249 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:49.743973 21250 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:49.744086 21220 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:49.744518 17661 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:49.744720 17530 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:49.744910 21220 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:49.745573 21221 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:49.747398 21221 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.748282 17530 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.748328 18057 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.749922 17266 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.750015 17398 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.750576 17661 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.751593 19505 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.751986 17120 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.755537 17791 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ab18888a9724446b9ee4fb9efe811b09 (127.0.99.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.759820 17924 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } }
I20240503 08:00:49.761911 17530 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:49.762063 19505 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:49.762233 17398 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:49.762560 21248 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:49.762645 18057 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:49.762825 17266 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:49.763092 21248 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:49.763298 21220 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:49.763342 17661 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:49.763422 21248 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:49.763541 21221 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:49.763705 21247 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:49.765985 21220 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.767396 17109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab988cc96fc940239e18f54ec9c9b401 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:49.767865 19505 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.768282 17398 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.769309 17266 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.769963 18057 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.770313 17119 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:49.771668 17661 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.771797 17530 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } }
I20240503 08:00:49.774168 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet ab988cc96fc940239e18f54ec9c9b401 with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:49.775625 21273 tablet_replica.cc:330] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:49.775805 21273 raft_consensus.cc:2229] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:49.776237 21273 raft_consensus.cc:2258] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:49.777118 21273 ts_tablet_manager.cc:1875] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:49.778888 21273 ts_tablet_manager.cc:1888] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:49.778946 21273 log.cc:1196] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ab988cc96fc940239e18f54ec9c9b401
I20240503 08:00:49.779388 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet ab988cc96fc940239e18f54ec9c9b401 (table rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]) successfully deleted
I20240503 08:00:50.211994 21244 raft_consensus.cc:1050] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:50.212455 21244 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:50.214095 17266 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:50.214311 17398 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:50.214587 17924 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:50.214733 19636 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:50.214733 17791 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:50.214754 18057 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:50.214958 21185 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:50.215152 21184 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:50.215270 17661 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:00:50.215472 21185 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:50.215600 21244 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:50.215912 21244 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:50.216095 21184 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:50.216260 21285 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:00:50.218824 21184 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.220468 18057 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.220600 17398 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.221753 17124 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:50.222273 19636 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.222554 17924 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.222883 17266 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.227440 17661 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.227558 17791 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.231834 17530 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:50.233229 17398 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:50.233265 17266 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:50.233227 17661 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:50.233623 19636 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:50.233726 21285 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:50.233945 21288 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:50.234078 17924 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:50.234090 21184 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:50.234390 18057 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:00:50.234522 21185 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:50.234966 21288 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:50.235096 21185 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:00:50.236868 21288 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.237684 17398 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.237704 18057 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.238461 17924 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.239409 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10e598a3a124472eab35c97689057ace with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:50.239312 19636 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.239817 17662 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.240139 17266 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.242506 17115 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 6 to 7, VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:50.245038 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e598a3a124472eab35c97689057ace with delete_type TABLET_DATA_TOMBSTONED (TS 84305d92b082406380b107114e4cd61b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57970
I20240503 08:00:50.245512 21273 tablet_replica.cc:330] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: stopping tablet replica
I20240503 08:00:50.245661 21273 raft_consensus.cc:2229] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:50.245865 21273 raft_consensus.cc:2258] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:50.246601 21273 ts_tablet_manager.cc:1875] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:50.248414 21273 ts_tablet_manager.cc:1888] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:00:50.248462 21273 log.cc:1196] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/10e598a3a124472eab35c97689057ace
I20240503 08:00:50.248849 17110 catalog_manager.cc:4830] TS 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): tablet 10e598a3a124472eab35c97689057ace (table rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]) successfully deleted
I20240503 08:00:50.681249 17905 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.683298 19354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:50.688692 17640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.699949 17510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.701207 19485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:50.702924 18037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.713691 17246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.714823 19610 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:50.730518 17377 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:50.735491 17774 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.841231 19354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:50.841935 17640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.844501 17905 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.864439 17377 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:50.866653 17774 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.869946 17246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:50.876590 19610 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:50.884084 18037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:50.885265 17510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:50.899667 19485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:50.940238 17397 consensus_queue.cc:237] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:50.942169 17266 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.942169 18057 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.942281 17530 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.942404 17791 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.942800 21399 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.943035 21398 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.943135 17924 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.943209 21133 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.943737 21212 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.943863 21398 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.944612 19505 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:50.944720 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8500b198108649a7bcb614c93cf53a3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:50.945678 21133 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.946519 21133 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.946817 17530 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.947146 18057 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.947870 19505 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.948882 17791 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.951056 17117 catalog_manager.cc:5484] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.951889 17924 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.955415 17266 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.964018 17397 consensus_queue.cc:237] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:50.965821 19505 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.966117 17530 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.966229 17266 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.966452 18057 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.966491 21133 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.966925 17924 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.967659 21133 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.968513 21133 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.969084 21418 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.969202 17791 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.969542 21133 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.969830 21133 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.971873 21398 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:50.972254 17791 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:50.974488 18057 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:50.975193 17121 catalog_manager.cc:5484] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b reported cstate change: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
W20240503 08:00:50.976573 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d1de048a83e45548afac96309abf4d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:50.977114 17924 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:50.979277 19505 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:50.981149 17266 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:50.983992 17530 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:50.988772 17530 consensus_queue.cc:237] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:50.990350 18057 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.990361 19636 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.990420 17266 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.990736 21288 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:00:50.990923 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5154e7f64d1542979f9363a429fa469c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:50.990927 21185 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.991151 17397 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.991762 21285 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.992676 21184 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.993000 17924 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.993193 17791 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:50.993402 21184 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.993973 21184 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:50.995254 21185 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.995755 17397 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.995790 18057 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.995836 17924 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.997802 19636 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:50.998690 17115 catalog_manager.cc:5484] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.000172 17266 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.002903 17793 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.010288 17397 consensus_queue.cc:237] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.012033 19505 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.012292 18057 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.012455 21406 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.012609 17529 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:51.012872 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42be6135d7654954b6886c32dfc688c5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:51.012928 17661 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.013381 21398 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.013677 17266 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.014012 21398 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.014161 21406 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.015084 21406 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.017441 17793 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.017390 21398 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.017798 21133 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.018827 19505 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.019152 17793 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.019670 18057 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.020210 17266 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.021363 17661 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.022687 17124 catalog_manager.cc:5484] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 reported cstate change: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.027194 17529 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.033970 17924 consensus_queue.cc:237] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.035946 19636 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:51.036015 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15a5f02e61d24029b0a5757556085733. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:51.036172 18057 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.036289 21459 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.036537 17661 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.036716 21463 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.036870 17529 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.036917 21462 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.037214 17397 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.037532 17266 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.038161 21462 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.038296 21459 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.038447 21461 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.039847 18057 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.040063 21460 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.040721 17397 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.040814 17266 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.042709 17121 catalog_manager.cc:5484] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.042716 19636 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.045476 17529 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.046067 17661 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.054025 17924 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
W20240503 08:00:51.056267 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab988cc96fc940239e18f54ec9c9b401. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:51.056304 19505 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.056622 18057 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.056936 17529 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.057085 17397 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.057094 21460 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.057317 21459 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.057449 21463 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.057869 21462 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.058051 17266 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.058501 17661 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.058902 21459 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.059312 21459 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.060395 21459 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.060868 17397 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.062249 19505 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.063696 17661 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.064478 18057 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.065742 17115 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.067248 17529 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.071290 17266 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.076371 17924 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.078091 17661 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:51.078343 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e57bf40798824baf8050988874664154. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:51.078720 17529 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.078742 18057 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.078742 19505 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.078953 17397 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.079320 21476 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.079455 17266 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.079918 21460 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.080094 21249 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.080258 21462 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.080405 21476 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.080540 21461 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.082746 21461 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.083560 17265 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.083595 18057 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.085700 17115 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:51.085876 17397 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.086575 17529 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.086601 17661 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.094447 19505 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.098421 17529 consensus_queue.cc:237] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
W20240503 08:00:51.100214 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a79ddd7d381c4ccbbd1a1e7d3ec8366f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:51.100234 17265 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.100252 19374 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.100804 17924 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.100821 18057 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.100855 17793 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.101099 17397 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.101191 21288 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.101382 21439 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.101501 21285 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.101665 21501 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.101848 21185 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.102035 21184 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.104610 21185 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.105131 19374 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.105198 17793 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.105188 17265 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.107753 17397 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.108749 17117 catalog_manager.cc:5484] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:51.110090 18057 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.110868 17924 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.125901 17529 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.127924 19636 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.128154 18057 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.128444 17661 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.128472 17265 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.128592 17924 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.128964 21184 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.129287 17397 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:51.129717 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10e598a3a124472eab35c97689057ace. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:51.129745 21185 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.129926 21514 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.130098 21439 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.130354 21439 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.130610 21439 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.132313 21501 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.132750 17924 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.132856 17661 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.133486 19636 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.135445 17124 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:51.137319 18057 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.137919 17397 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.138065 17265 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:51.146783 17793 consensus_queue.cc:237] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.148666 18057 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.149024 17924 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.149080 19505 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.149027 20813 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.149407 21521 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.149569 20813 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.150152 17529 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.150341 17397 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.150481 17265 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:51.150755 21521 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.150893 21524 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.151048 20813 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:51.152042 21523 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.152401 17924 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.155041 19505 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.155673 17124 catalog_manager.cc:5484] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "84305d92b082406380b107114e4cd61b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.157253 18057 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.158447 17397 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
W20240503 08:00:51.159101 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbe0edacc8ce4ba8be5e533c82ea92e0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:51.160634 17265 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.161285 17529 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:51.320878 21582 ts_tablet_manager.cc:902] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:51.321383 21582 tablet_copy_client.cc:308] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:51.321875 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet 8500b198108649a7bcb614c93cf53a3a from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:50179)
I20240503 08:00:51.321995 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet 8500b198108649a7bcb614c93cf53a3a from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:50179: session id = 0665f4f85b984a8bba213ad479a88bad-8500b198108649a7bcb614c93cf53a3a
I20240503 08:00:51.322849 17418 tablet_copy_source_session.cc:205] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.323663 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.325430 21582 tablet_copy_client.cc:783] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.325574 21582 tablet_copy_client.cc:647] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.326581 21582 tablet_copy_client.cc:523] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.327484 21582 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:51.347126 21582 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.347389 21582 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:51.347543 21582 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.001s
I20240503 08:00:51.347941 21582 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.348089 21582 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:51.348198 21582 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.348482 21582 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:51.348974 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-8500b198108649a7bcb614c93cf53a3a received from {username='slave'} at 127.0.99.137:50179
I20240503 08:00:51.349082 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-8500b198108649a7bcb614c93cf53a3a on tablet 8500b198108649a7bcb614c93cf53a3a with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:51.475741 21638 ts_tablet_manager.cc:902] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:51.476230 21638 tablet_copy_client.cc:308] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:51.479966 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet 3d1de048a83e45548afac96309abf4d2 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:39241)
I20240503 08:00:51.480074 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet 3d1de048a83e45548afac96309abf4d2 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:39241: session id = b65378a282b9443484843de0b8906df9-3d1de048a83e45548afac96309abf4d2
I20240503 08:00:51.480943 17418 tablet_copy_source_session.cc:205] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.481740 21638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1de048a83e45548afac96309abf4d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.483402 21638 tablet_copy_client.cc:783] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.483551 21638 tablet_copy_client.cc:647] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.484547 21638 tablet_copy_client.cc:523] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.485499 21638 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:51.503567 21638 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.503821 21638 tablet_bootstrap.cc:492] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:51.503970 21638 ts_tablet_manager.cc:1367] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20240503 08:00:51.504344 21638 raft_consensus.cc:348] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.504462 21638 raft_consensus.cc:729] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:51.504540 21638 consensus_queue.cc:260] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.504789 21638 ts_tablet_manager.cc:1398] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:51.505348 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session b65378a282b9443484843de0b8906df9-3d1de048a83e45548afac96309abf4d2 received from {username='slave'} at 127.0.99.136:39241
I20240503 08:00:51.505437 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session b65378a282b9443484843de0b8906df9-3d1de048a83e45548afac96309abf4d2 on tablet 3d1de048a83e45548afac96309abf4d2 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:51.512759 21582 ts_tablet_manager.cc:902] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:51.513173 21582 tablet_copy_client.cc:308] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39647
I20240503 08:00:51.513567 17550 tablet_copy_service.cc:140] P 7af8f2a481294a2cb93e5761947978cc: Received BeginTabletCopySession request for tablet 5154e7f64d1542979f9363a429fa469c from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:40241)
I20240503 08:00:51.513661 17550 tablet_copy_service.cc:161] P 7af8f2a481294a2cb93e5761947978cc: Beginning new tablet copy session on tablet 5154e7f64d1542979f9363a429fa469c from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:40241: session id = 0665f4f85b984a8bba213ad479a88bad-5154e7f64d1542979f9363a429fa469c
I20240503 08:00:51.514314 17550 tablet_copy_source_session.cc:205] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.515048 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154e7f64d1542979f9363a429fa469c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.516608 21582 tablet_copy_client.cc:783] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.516745 21582 tablet_copy_client.cc:647] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.517694 21582 tablet_copy_client.cc:523] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.518579 21582 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:51.526598 21638 ts_tablet_manager.cc:902] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:51.527041 21638 tablet_copy_client.cc:308] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:51.527508 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet ab988cc96fc940239e18f54ec9c9b401 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:37051)
I20240503 08:00:51.527629 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet ab988cc96fc940239e18f54ec9c9b401 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:37051: session id = b65378a282b9443484843de0b8906df9-ab988cc96fc940239e18f54ec9c9b401
I20240503 08:00:51.528539 17946 tablet_copy_source_session.cc:205] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.529517 21638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.531302 21638 tablet_copy_client.cc:783] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.531430 21638 tablet_copy_client.cc:647] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.532487 21638 tablet_copy_client.cc:523] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.533389 21638 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:51.539666 21582 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.539894 21582 tablet_bootstrap.cc:492] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:51.540041 21582 ts_tablet_manager.cc:1367] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.004s
I20240503 08:00:51.540437 21582 raft_consensus.cc:348] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.540545 21582 raft_consensus.cc:729] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:51.540637 21582 consensus_queue.cc:260] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.540882 21582 ts_tablet_manager.cc:1398] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:51.541425 17550 tablet_copy_service.cc:342] P 7af8f2a481294a2cb93e5761947978cc: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-5154e7f64d1542979f9363a429fa469c received from {username='slave'} at 127.0.99.137:40241
I20240503 08:00:51.541512 17550 tablet_copy_service.cc:433] P 7af8f2a481294a2cb93e5761947978cc: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-5154e7f64d1542979f9363a429fa469c on tablet 5154e7f64d1542979f9363a429fa469c with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:51.554068 21638 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.581759 21582 ts_tablet_manager.cc:902] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:51.582194 21582 tablet_copy_client.cc:308] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:51.582633 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet e57bf40798824baf8050988874664154 from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:47395)
I20240503 08:00:51.582741 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet e57bf40798824baf8050988874664154 from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:47395: session id = 0665f4f85b984a8bba213ad479a88bad-e57bf40798824baf8050988874664154
I20240503 08:00:51.583421 17946 tablet_copy_source_session.cc:205] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.584174 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.585228 21638 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:51.585426 21638 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.052s user 0.021s sys 0.000s
I20240503 08:00:51.585840 21638 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.585955 21638 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:51.585983 21582 tablet_copy_client.cc:783] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.586071 21638 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.586197 21582 tablet_copy_client.cc:647] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.586587 21638 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:51.587235 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session b65378a282b9443484843de0b8906df9-ab988cc96fc940239e18f54ec9c9b401 received from {username='slave'} at 127.0.99.136:37051
I20240503 08:00:51.587306 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session b65378a282b9443484843de0b8906df9-ab988cc96fc940239e18f54ec9c9b401 on tablet ab988cc96fc940239e18f54ec9c9b401 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:51.587471 21582 tablet_copy_client.cc:523] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.588734 21582 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:51.607110 21582 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.607331 21582 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:51.607420 21662 ts_tablet_manager.cc:902] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:51.607491 21582 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240503 08:00:51.607921 21662 tablet_copy_client.cc:308] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39647
I20240503 08:00:51.607933 21582 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.608106 21582 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:51.608218 21582 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.608827 17550 tablet_copy_service.cc:140] P 7af8f2a481294a2cb93e5761947978cc: Received BeginTabletCopySession request for tablet 10e598a3a124472eab35c97689057ace from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:40241)
I20240503 08:00:51.608930 17550 tablet_copy_service.cc:161] P 7af8f2a481294a2cb93e5761947978cc: Beginning new tablet copy session on tablet 10e598a3a124472eab35c97689057ace from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:40241: session id = 0665f4f85b984a8bba213ad479a88bad-10e598a3a124472eab35c97689057ace
I20240503 08:00:51.609119 21582 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:00:51.609841 17550 tablet_copy_source_session.cc:205] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.610497 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-e57bf40798824baf8050988874664154 received from {username='slave'} at 127.0.99.137:47395
I20240503 08:00:51.610574 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-e57bf40798824baf8050988874664154 on tablet e57bf40798824baf8050988874664154 with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:51.611088 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.612588 21638 ts_tablet_manager.cc:902] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:51.612993 21638 tablet_copy_client.cc:308] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39647
I20240503 08:00:51.613207 21662 tablet_copy_client.cc:783] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.613397 17550 tablet_copy_service.cc:140] P 7af8f2a481294a2cb93e5761947978cc: Received BeginTabletCopySession request for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:34313)
I20240503 08:00:51.613494 17550 tablet_copy_service.cc:161] P 7af8f2a481294a2cb93e5761947978cc: Beginning new tablet copy session on tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:34313: session id = b65378a282b9443484843de0b8906df9-a79ddd7d381c4ccbbd1a1e7d3ec8366f
I20240503 08:00:51.614251 17550 tablet_copy_source_session.cc:205] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.614789 21662 tablet_copy_client.cc:647] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.615016 21638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.615765 21662 tablet_copy_client.cc:523] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.616892 21638 tablet_copy_client.cc:783] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.617017 21638 tablet_copy_client.cc:647] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.617566 21662 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:51.621783 21638 tablet_copy_client.cc:523] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.622710 21638 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:51.629253 21673 ts_tablet_manager.cc:902] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:51.629899 21673 tablet_copy_client.cc:308] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:51.633458 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet 42be6135d7654954b6886c32dfc688c5 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:39241)
I20240503 08:00:51.633562 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet 42be6135d7654954b6886c32dfc688c5 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:39241: session id = b65378a282b9443484843de0b8906df9-42be6135d7654954b6886c32dfc688c5
I20240503 08:00:51.634382 17418 tablet_copy_source_session.cc:205] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.635136 21673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be6135d7654954b6886c32dfc688c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.637336 21673 tablet_copy_client.cc:783] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.637492 21673 tablet_copy_client.cc:647] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.638474 21673 tablet_copy_client.cc:523] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.639519 21673 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:51.644905 21582 ts_tablet_manager.cc:902] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:51.644939 21679 ts_tablet_manager.cc:902] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825)
I20240503 08:00:51.645340 21582 tablet_copy_client.cc:308] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:51.645416 21679 tablet_copy_client.cc:308] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:38825
I20240503 08:00:51.646278 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet 15a5f02e61d24029b0a5757556085733 from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:47395)
I20240503 08:00:51.646382 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet 15a5f02e61d24029b0a5757556085733 from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:47395: session id = 0665f4f85b984a8bba213ad479a88bad-15a5f02e61d24029b0a5757556085733
I20240503 08:00:51.647200 17946 tablet_copy_source_session.cc:205] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.648485 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15a5f02e61d24029b0a5757556085733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.650293 21638 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.650525 21638 tablet_bootstrap.cc:492] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:51.650693 21638 ts_tablet_manager.cc:1367] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.001s
I20240503 08:00:51.651119 21638 raft_consensus.cc:348] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.651247 21638 raft_consensus.cc:729] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:51.651341 21638 consensus_queue.cc:260] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.651624 21638 ts_tablet_manager.cc:1398] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:51.652060 17550 tablet_copy_service.cc:342] P 7af8f2a481294a2cb93e5761947978cc: Request end of tablet copy session b65378a282b9443484843de0b8906df9-a79ddd7d381c4ccbbd1a1e7d3ec8366f received from {username='slave'} at 127.0.99.136:34313
I20240503 08:00:51.652133 17550 tablet_copy_service.cc:433] P 7af8f2a481294a2cb93e5761947978cc: ending tablet copy session b65378a282b9443484843de0b8906df9-a79ddd7d381c4ccbbd1a1e7d3ec8366f on tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:51.655053 21582 tablet_copy_client.cc:783] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.655201 21582 tablet_copy_client.cc:647] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.658003 17814 tablet_copy_service.cc:140] P 84305d92b082406380b107114e4cd61b: Received BeginTabletCopySession request for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:60989)
I20240503 08:00:51.658114 17814 tablet_copy_service.cc:161] P 84305d92b082406380b107114e4cd61b: Beginning new tablet copy session on tablet bbe0edacc8ce4ba8be5e533c82ea92e0 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:60989: session id = b65378a282b9443484843de0b8906df9-bbe0edacc8ce4ba8be5e533c82ea92e0
I20240503 08:00:51.658895 17814 tablet_copy_source_session.cc:205] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:51.659294 21582 tablet_copy_client.cc:523] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.660408 21679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe0edacc8ce4ba8be5e533c82ea92e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:51.660454 21582 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:51.662552 21679 tablet_copy_client.cc:783] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:51.662874 21679 tablet_copy_client.cc:647] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:51.664132 21679 tablet_copy_client.cc:523] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:51.665505 21679 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:51.675642 21673 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.676017 21673 tablet_bootstrap.cc:492] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:51.676242 21673 ts_tablet_manager.cc:1367] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.000s
I20240503 08:00:51.676746 21673 raft_consensus.cc:348] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.676985 21673 raft_consensus.cc:729] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:51.677161 21673 consensus_queue.cc:260] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.678259 21673 ts_tablet_manager.cc:1398] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:00:51.679256 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session b65378a282b9443484843de0b8906df9-42be6135d7654954b6886c32dfc688c5 received from {username='slave'} at 127.0.99.136:39241
I20240503 08:00:51.679369 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session b65378a282b9443484843de0b8906df9-42be6135d7654954b6886c32dfc688c5 on tablet 42be6135d7654954b6886c32dfc688c5 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:51.687018 21662 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.687356 21662 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:51.687539 21662 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.070s user 0.023s sys 0.003s
I20240503 08:00:51.688016 21662 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.688167 21662 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:51.688289 21582 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.688280 21662 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.688565 21582 tablet_bootstrap.cc:492] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:51.688704 21662 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:00:51.688714 21582 ts_tablet_manager.cc:1367] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20240503 08:00:51.689272 21582 raft_consensus.cc:348] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.689399 21582 raft_consensus.cc:729] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:51.689499 21582 consensus_queue.cc:260] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:51.689756 21582 ts_tablet_manager.cc:1398] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:51.690140 17550 tablet_copy_service.cc:342] P 7af8f2a481294a2cb93e5761947978cc: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-10e598a3a124472eab35c97689057ace received from {username='slave'} at 127.0.99.137:40241
I20240503 08:00:51.690233 17550 tablet_copy_service.cc:433] P 7af8f2a481294a2cb93e5761947978cc: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-10e598a3a124472eab35c97689057ace on tablet 10e598a3a124472eab35c97689057ace with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:51.691038 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-15a5f02e61d24029b0a5757556085733 received from {username='slave'} at 127.0.99.137:47395
I20240503 08:00:51.691110 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-15a5f02e61d24029b0a5757556085733 on tablet 15a5f02e61d24029b0a5757556085733 with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:51.701745 21679 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:51.702046 21679 tablet_bootstrap.cc:492] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:51.702214 21679 ts_tablet_manager.cc:1367] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.037s user 0.025s sys 0.002s
I20240503 08:00:51.702652 21679 raft_consensus.cc:348] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.702816 21679 raft_consensus.cc:729] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:51.702956 21679 consensus_queue.cc:260] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:51.703282 21679 ts_tablet_manager.cc:1398] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:51.703815 17814 tablet_copy_service.cc:342] P 84305d92b082406380b107114e4cd61b: Request end of tablet copy session b65378a282b9443484843de0b8906df9-bbe0edacc8ce4ba8be5e533c82ea92e0 received from {username='slave'} at 127.0.99.136:60989
I20240503 08:00:51.703903 17814 tablet_copy_service.cc:433] P 84305d92b082406380b107114e4cd61b: ending tablet copy session b65378a282b9443484843de0b8906df9-bbe0edacc8ce4ba8be5e533c82ea92e0 on tablet bbe0edacc8ce4ba8be5e533c82ea92e0 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:51.812748 19374 raft_consensus.cc:1203] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:51.975911 19636 raft_consensus.cc:1203] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:51.987299 19374 raft_consensus.cc:1203] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:52.032328 19636 raft_consensus.cc:1203] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:52.067993 19636 raft_consensus.cc:1203] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:52.099020 19374 raft_consensus.cc:1203] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:52.105221 19636 raft_consensus.cc:1203] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:52.108062 19374 raft_consensus.cc:1203] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:52.116111 19636 raft_consensus.cc:1203] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:52.124256 19374 raft_consensus.cc:1203] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:52.379705 21660 raft_consensus.cc:1050] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:52.380316 21660 consensus_queue.cc:237] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.382483 18058 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.382515 17926 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.382639 17529 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.383034 17793 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.383246 19374 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.383271 19505 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.383489 21844 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.383738 21846 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.383910 21847 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.384002 17265 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.384060 21845 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.384312 21133 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.384438 21660 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.385015 21661 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.388025 21846 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.388911 17529 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.389101 19374 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.390568 18058 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.390862 19505 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.391762 17117 catalog_manager.cc:5484] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.393605 17265 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.394295 17926 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.396313 17397 consensus_queue.cc:237] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.396983 17793 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.398150 18058 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.398468 17791 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.398650 19505 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.399014 21133 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.399727 21133 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.400584 17926 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.400947 17265 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.401188 21660 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.401465 21845 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.401983 19374 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.402338 21845 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.402958 21660 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.405001 21661 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.407195 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8500b198108649a7bcb614c93cf53a3a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.407050 17791 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.407999 17265 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.409353 17117 catalog_manager.cc:5484] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b reported cstate change: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.409293 19373 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.411607 18058 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.412238 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet 8500b198108649a7bcb614c93cf53a3a with delete_type TABLET_DATA_TOMBSTONED (TS 7af8f2a481294a2cb93e5761947978cc not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36620
I20240503 08:00:52.412861 21860 tablet_replica.cc:330] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc: stopping tablet replica
I20240503 08:00:52.413193 21860 raft_consensus.cc:2229] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.413571 21860 raft_consensus.cc:2258] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.414320 19505 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.414358 17926 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.418418 21860 ts_tablet_manager.cc:1875] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.420557 21860 ts_tablet_manager.cc:1888] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.420627 21860 log.cc:1196] T 8500b198108649a7bcb614c93cf53a3a P 7af8f2a481294a2cb93e5761947978cc: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8500b198108649a7bcb614c93cf53a3a
I20240503 08:00:52.421141 17108 catalog_manager.cc:4830] TS 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647): tablet 8500b198108649a7bcb614c93cf53a3a (table rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]) successfully deleted
I20240503 08:00:52.429212 21791 raft_consensus.cc:1050] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:52.429674 21791 consensus_queue.cc:237] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.431630 17529 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.431666 19505 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.431740 17265 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.432021 17397 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.432022 19636 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.432314 21523 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.432364 18058 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.432488 21864 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.432652 21790 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.433147 21523 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.433291 21862 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.434623 21864 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.435254 17926 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.436005 21862 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.436470 21863 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.437106 21846 raft_consensus.cc:1050] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:52.437129 19505 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.437602 21846 consensus_queue.cc:237] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.438594 18058 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.439963 17124 catalog_manager.cc:5484] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 reported cstate change: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84305d92b082406380b107114e4cd61b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.440412 17529 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.440738 19636 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.440865 21845 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.441035 17265 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.441174 18058 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.441365 21661 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.441987 21845 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.441990 17397 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.442471 17791 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.442837 21845 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.443624 19505 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.444500 17926 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.444331 19636 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.444450 17530 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.446578 21845 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.447360 21661 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.447407 17265 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.449141 21661 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.449486 21133 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.450273 17924 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.451901 17529 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.452481 18058 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.452566 19635 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.453097 19505 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.454171 17791 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.456548 17120 catalog_manager.cc:5484] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 reported cstate change: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.458226 17924 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.459976 17791 consensus_queue.cc:237] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.460811 21285 raft_consensus.cc:1050] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:52.461335 21285 consensus_queue.cc:237] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.461777 17397 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.462060 19505 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.462198 18058 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.462477 21523 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.462740 19635 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.462807 21862 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.463553 17263 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.463616 18058 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.463939 21514 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.464102 19373 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.464277 21514 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.464375 17793 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.464440 21184 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.464838 17397 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.465376 21184 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.465664 21184 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.466173 19635 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.466523 21184 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.467831 17926 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.469218 21862 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.469473 17266 raft_consensus.cc:1261] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 84305d92b082406380b107114e4cd61b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.469868 21862 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.470391 21862 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.470638 21523 consensus_queue.cc:1035] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.472277 21864 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.472736 17926 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.473129 19505 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.474442 17110 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.474696 18057 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.476871 17398 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.477627 17924 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.477805 17121 catalog_manager.cc:5484] T bbe0edacc8ce4ba8be5e533c82ea92e0 P ab18888a9724446b9ee4fb9efe811b09 reported cstate change: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "84305d92b082406380b107114e4cd61b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.478075 21887 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 08:00:52.480072 21285 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.480540 17793 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.481391 17265 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.482219 17263 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.482666 19373 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.485004 17120 catalog_manager.cc:5484] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad reported cstate change: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.485045 17924 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.485226 17397 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.487031 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe0edacc8ce4ba8be5e533c82ea92e0 with delete_type TABLET_DATA_TOMBSTONED (TS 7af8f2a481294a2cb93e5761947978cc not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36620
I20240503 08:00:52.487411 19636 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.487679 21860 tablet_replica.cc:330] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc: stopping tablet replica
I20240503 08:00:52.487852 21860 raft_consensus.cc:2229] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.488111 21860 raft_consensus.cc:2258] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.489893 21860 ts_tablet_manager.cc:1875] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.490775 19635 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.491902 21860 ts_tablet_manager.cc:1888] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.491966 21860 log.cc:1196] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 7af8f2a481294a2cb93e5761947978cc: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/bbe0edacc8ce4ba8be5e533c82ea92e0
I20240503 08:00:52.491806 18057 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.493734 17266 raft_consensus.cc:2932] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.493930 17108 catalog_manager.cc:4830] TS 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647): tablet bbe0edacc8ce4ba8be5e533c82ea92e0 (table rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]) successfully deleted
I20240503 08:00:52.494606 21846 raft_consensus.cc:1050] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:52.495101 21846 consensus_queue.cc:237] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.495891 17529 consensus_queue.cc:237] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.496932 18057 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.497331 21660 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.498257 17661 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.498592 17263 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.498617 19373 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.498939 19635 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.498937 19505 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.499166 21514 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.499300 21845 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.499459 21846 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.499756 21846 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.499800 18058 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.500211 21184 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.500286 19636 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.500507 21184 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.501121 17791 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.501482 17793 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.501842 21285 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.502161 21285 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.502840 17266 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.503348 21660 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.503401 17924 raft_consensus.cc:1261] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.503701 17529 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.503871 21514 consensus_queue.cc:1035] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.505041 21845 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.505223 21660 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.504859 21888 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.505291 19373 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.507390 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5154e7f64d1542979f9363a429fa469c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.508143 18058 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.508709 17121 catalog_manager.cc:5484] T 5154e7f64d1542979f9363a429fa469c P 0665f4f85b984a8bba213ad479a88bad reported cstate change: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.508886 19635 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.509905 17263 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.512238 17377 tablet_service.cc:1514] Processing DeleteTablet for tablet 5154e7f64d1542979f9363a429fa469c with delete_type TABLET_DATA_TOMBSTONED (TS f0a940e3a6e340a69ec7ac31d26c9298 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39992
I20240503 08:00:52.513792 17793 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.513559 21908 tablet_replica.cc:330] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298: stopping tablet replica
I20240503 08:00:52.514706 21908 raft_consensus.cc:2229] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.514982 21908 raft_consensus.cc:2258] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.515645 21845 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.516218 21908 ts_tablet_manager.cc:1875] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.518247 17661 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.518445 19504 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.519384 18057 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.521342 17529 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.521425 17791 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.522855 17924 raft_consensus.cc:2932] T 5154e7f64d1542979f9363a429fa469c P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.523766 17266 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.524436 19636 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.518260 17117 catalog_manager.cc:5484] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:52.530491 21908 ts_tablet_manager.cc:1888] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.530580 21908 log.cc:1196] T 5154e7f64d1542979f9363a429fa469c P f0a940e3a6e340a69ec7ac31d26c9298: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5154e7f64d1542979f9363a429fa469c
I20240503 08:00:52.531925 17108 catalog_manager.cc:4830] TS f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225): tablet 5154e7f64d1542979f9363a429fa469c (table rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]) successfully deleted
I20240503 08:00:52.537271 17397 consensus_queue.cc:237] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.538939 19636 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.539273 18057 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.539407 21847 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.539655 17661 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.540030 19504 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.540016 21845 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.540361 17266 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.540352 21845 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.540652 21844 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.540812 21845 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.541870 17791 raft_consensus.cc:1261] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.542658 21847 consensus_queue.cc:1035] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.543148 21844 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.543457 19636 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.544795 19504 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.545795 17121 catalog_manager.cc:5484] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:52.546257 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 42be6135d7654954b6886c32dfc688c5 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.547125 18057 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.548996 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet 42be6135d7654954b6886c32dfc688c5 with delete_type TABLET_DATA_TOMBSTONED (TS 7af8f2a481294a2cb93e5761947978cc not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36620
I20240503 08:00:52.549227 21860 tablet_replica.cc:330] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: stopping tablet replica
I20240503 08:00:52.549362 21860 raft_consensus.cc:2229] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.549577 21860 raft_consensus.cc:2258] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.550292 21860 ts_tablet_manager.cc:1875] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.551443 17661 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.551800 17266 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.552137 21860 ts_tablet_manager.cc:1888] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.552197 21860 log.cc:1196] T 42be6135d7654954b6886c32dfc688c5 P 7af8f2a481294a2cb93e5761947978cc: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/42be6135d7654954b6886c32dfc688c5
I20240503 08:00:52.552570 17791 raft_consensus.cc:2932] T 42be6135d7654954b6886c32dfc688c5 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.554116 17108 catalog_manager.cc:4830] TS 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647): tablet 42be6135d7654954b6886c32dfc688c5 (table rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]) successfully deleted
I20240503 08:00:52.588485 21888 raft_consensus.cc:1050] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:52.589046 21888 consensus_queue.cc:237] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.591101 17266 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.591207 17397 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.591235 17791 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.591610 21285 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.591897 18057 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.591964 17924 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.592466 21285 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.592603 21514 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.592952 21514 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.593578 19373 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.594151 19636 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.595459 21285 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.595608 21898 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.596161 21886 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.597877 21285 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.599534 19373 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.599779 19635 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.599778 17397 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.600143 17924 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.602427 18057 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.602496 17123 catalog_manager.cc:5484] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad reported cstate change: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.605640 17791 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.605736 17266 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.613579 17529 consensus_queue.cc:237] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.615226 17397 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.615406 17924 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.615594 17791 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.615594 21184 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.616322 21184 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.616581 19373 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.616653 21886 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.616945 21886 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.617322 18057 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.617551 17266 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:52.618765 19635 raft_consensus.cc:1261] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.618836 21886 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.619689 19504 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.620112 21431 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.620165 17530 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.620311 18056 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.620368 21887 consensus_queue.cc:1035] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.621148 17398 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.621786 17661 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.622088 17791 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.621836 21285 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.622740 21431 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.623698 21431 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.624547 21431 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.627301 21431 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.629040 21431 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.628919 19373 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
W20240503 08:00:52.629953 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a03fdbe7f824674a351fd4598dc2c63. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:52.630139 17924 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.630888 19635 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.631711 21792 raft_consensus.cc:1050] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:52.632198 21792 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.634449 18057 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.634783 21792 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.634704 17115 catalog_manager.cc:5484] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 0665f4f85b984a8bba213ad479a88bad reported cstate change: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.634912 19504 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.635578 17662 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.635658 17398 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.636029 21792 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.636611 17266 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.636807 17529 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.637105 21792 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.637414 21792 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.637755 21792 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.638988 21792 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.639556 17246 tablet_service.cc:1514] Processing DeleteTablet for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f with delete_type TABLET_DATA_TOMBSTONED (TS 8eb6822290b046f88c861159aa9421ec not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44958
I20240503 08:00:52.640131 21995 tablet_replica.cc:330] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec: stopping tablet replica
I20240503 08:00:52.640487 21995 raft_consensus.cc:2229] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.640877 21995 raft_consensus.cc:2258] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.641085 19635 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.642081 21995 ts_tablet_manager.cc:1875] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.642117 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.642705 18056 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.643003 17793 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.644160 21792 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.647681 21995 ts_tablet_manager.cc:1888] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.647771 21995 log.cc:1196] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 8eb6822290b046f88c861159aa9421ec: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a79ddd7d381c4ccbbd1a1e7d3ec8366f
I20240503 08:00:52.648522 17397 raft_consensus.cc:2932] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.649355 17108 catalog_manager.cc:4830] TS 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199): tablet a79ddd7d381c4ccbbd1a1e7d3ec8366f (table rebalance_test_table_2 [id=ae579ad3a39f469baee9c41e4f901ef0]) successfully deleted
I20240503 08:00:52.650399 21792 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.652722 17397 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.652792 17119 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:52.652961 19635 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.653247 17662 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.654953 17266 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.655162 21969 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:52.656097 17530 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:52.658538 18057 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.660064 21887 raft_consensus.cc:1050] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:52.652952 17529 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.660565 21887 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.660573 19505 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:52.663270 17924 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.663585 19373 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.663600 17397 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.664160 21886 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.664201 17661 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.664249 19635 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.664499 21886 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.664891 21898 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.665208 21898 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.665905 17397 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:52.666275 21886 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.666553 17115 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:52.667944 17662 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:52.668381 18057 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.668726 21886 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.673198 17266 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.673641 21285 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.673508 18056 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:52.674995 21285 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.675540 18056 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.676823 19373 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.679102 17119 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.679519 19635 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.679800 17926 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.681984 21979 raft_consensus.cc:1050] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:52.682353 17662 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.682502 21979 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.684578 19505 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.684795 18056 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.685231 17265 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.685262 17398 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.685681 21792 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.686079 21792 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.686120 17530 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.686371 21792 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.686700 21792 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.686900 21988 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.687836 17662 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.689913 19373 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.690586 21979 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.690739 21984 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.691310 17266 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.692744 17529 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.694799 17926 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.694890 18057 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.695183 19373 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.695297 21886 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.695470 19635 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.695585 21888 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.695768 17661 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.695966 21888 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.696578 19504 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.697037 21888 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.697243 21886 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.697650 17266 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.697950 21285 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.698917 21285 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.699609 18057 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.701182 17124 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:52.702065 17926 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.703661 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10e598a3a124472eab35c97689057ace with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.703857 17661 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.703976 17791 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:52.704845 17926 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.706336 19635 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.707839 21979 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.710026 19374 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.710302 17265 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.710332 17377 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e598a3a124472eab35c97689057ace with delete_type TABLET_DATA_TOMBSTONED (TS f0a940e3a6e340a69ec7ac31d26c9298 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39992
I20240503 08:00:52.710605 17397 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.711393 21908 tablet_replica.cc:330] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: stopping tablet replica
I20240503 08:00:52.711517 21908 raft_consensus.cc:2229] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.711710 21908 pending_rounds.cc:62] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: Trying to abort 1 pending ops.
I20240503 08:00:52.711776 21908 pending_rounds.cc:68] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7023506442896113664 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "10e598a3a124472eab35c97689057ace" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } } }
I20240503 08:00:52.711934 21908 raft_consensus.cc:2894] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20240503 08:00:52.711995 21908 raft_consensus.cc:2866] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:00:52.712038 21908 raft_consensus.cc:2258] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.712611 19505 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.713143 17662 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.714006 21908 ts_tablet_manager.cc:1875] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.714244 18057 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.712893 19504 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.714687 17265 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.715902 21908 ts_tablet_manager.cc:1888] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.715963 21908 log.cc:1196] T 10e598a3a124472eab35c97689057ace P f0a940e3a6e340a69ec7ac31d26c9298: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/10e598a3a124472eab35c97689057ace
I20240503 08:00:52.716172 21979 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.716435 17108 catalog_manager.cc:4830] TS f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225): tablet 10e598a3a124472eab35c97689057ace (table rebalance_test_table_3 [id=bb56333199224f02a5315e8390c33c82]) successfully deleted
I20240503 08:00:52.716579 18056 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.717132 17662 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.717494 21979 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.716259 17119 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.718899 21979 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.719223 17530 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.719614 19374 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.719839 21792 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.720335 21792 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.721961 19635 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.723333 21986 raft_consensus.cc:1050] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:52.722343 21989 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.723424 17530 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.723841 21986 consensus_queue.cc:237] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.725728 19636 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.725728 17661 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.726069 21989 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.726167 19374 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.726425 21989 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.726537 17266 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.726868 18056 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.726897 17397 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.727027 17530 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:52.727571 21989 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.728144 21981 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.728602 21981 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.728643 17530 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.728885 21981 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.729164 21981 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:52.729341 21989 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.730057 19636 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.731472 17662 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.732407 17109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab988cc96fc940239e18f54ec9c9b401 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.733630 17120 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.733662 19504 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.733695 18057 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.735989 17266 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.723850 17926 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.736277 21986 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.737717 19374 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.737865 17377 tablet_service.cc:1514] Processing DeleteTablet for tablet ab988cc96fc940239e18f54ec9c9b401 with delete_type TABLET_DATA_TOMBSTONED (TS f0a940e3a6e340a69ec7ac31d26c9298 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39992
I20240503 08:00:52.738113 21908 tablet_replica.cc:330] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298: stopping tablet replica
I20240503 08:00:52.738196 21792 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.738248 21908 raft_consensus.cc:2229] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.738466 21908 raft_consensus.cc:2258] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.739187 21908 ts_tablet_manager.cc:1875] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.739315 19504 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.739645 21979 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.740506 19373 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.742980 17119 catalog_manager.cc:5484] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:52.743212 21908 ts_tablet_manager.cc:1888] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.743559 18057 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.743854 21908 log.cc:1196] T ab988cc96fc940239e18f54ec9c9b401 P f0a940e3a6e340a69ec7ac31d26c9298: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ab988cc96fc940239e18f54ec9c9b401
I20240503 08:00:52.743868 17530 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.744216 17265 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.744378 17662 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.744506 21792 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.744601 17398 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.744792 17108 catalog_manager.cc:4830] TS f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225): tablet ab988cc96fc940239e18f54ec9c9b401 (table rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]) successfully deleted
I20240503 08:00:52.744804 21792 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.745323 17397 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.747160 21792 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.747520 21792 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.748713 17266 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.749451 21987 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.749715 17265 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.751325 17109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e57bf40798824baf8050988874664154 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.752142 17397 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.752180 17263 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.752241 17662 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.754554 19373 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.755110 19504 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.756160 17115 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.756845 18057 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.758724 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet e57bf40798824baf8050988874664154 with delete_type TABLET_DATA_TOMBSTONED (TS 7af8f2a481294a2cb93e5761947978cc not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36620
I20240503 08:00:52.758919 21860 tablet_replica.cc:330] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: stopping tablet replica
I20240503 08:00:52.759045 21860 raft_consensus.cc:2229] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.759270 21860 raft_consensus.cc:2258] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.759975 21860 ts_tablet_manager.cc:1875] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.761807 17661 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.762009 21860 ts_tablet_manager.cc:1888] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.762068 21860 log.cc:1196] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e57bf40798824baf8050988874664154
I20240503 08:00:52.762358 18056 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.762576 17108 catalog_manager.cc:4830] TS 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647): tablet e57bf40798824baf8050988874664154 (table rebalance_test_table_6 [id=c3f019c7b40640179b58fcf15dbc7263]) successfully deleted
I20240503 08:00:52.764250 19635 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.766961 18057 consensus_queue.cc:237] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:52.768019 17926 consensus_queue.cc:237] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:52.768851 17263 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.769284 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.769344 19635 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.769819 19635 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.770067 17530 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.770149 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.770174 17661 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.770407 21984 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.770515 17397 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.771149 18056 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.771466 19373 raft_consensus.cc:1261] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.771534 22021 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.771781 21987 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.771924 21984 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.772233 21988 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.772404 21989 consensus_queue.cc:1035] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.773384 17529 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.773772 22031 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.774497 21989 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.774909 17661 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.775388 18056 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.775550 17662 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.775892 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.776145 17109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 15a5f02e61d24029b0a5757556085733 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.776397 17397 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.776676 17926 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:52.776947 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.776950 19635 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.777259 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:52.777518 17397 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.778214 19373 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.778338 17119 catalog_manager.cc:5484] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:52.782013 17529 raft_consensus.cc:2932] T 15a5f02e61d24029b0a5757556085733 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:52.783520 17661 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:52.782994 22036 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:52.786536 17246 tablet_service.cc:1514] Processing DeleteTablet for tablet 15a5f02e61d24029b0a5757556085733 with delete_type TABLET_DATA_TOMBSTONED (TS 8eb6822290b046f88c861159aa9421ec not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44958
I20240503 08:00:52.787108 21995 tablet_replica.cc:330] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec: stopping tablet replica
I20240503 08:00:52.787266 21995 raft_consensus.cc:2229] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.787506 21995 raft_consensus.cc:2258] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.787712 17121 catalog_manager.cc:5484] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:52.787796 19635 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:52.788540 21995 ts_tablet_manager.cc:1875] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.790213 17263 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:52.791657 21995 ts_tablet_manager.cc:1888] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.791734 21995 log.cc:1196] T 15a5f02e61d24029b0a5757556085733 P 8eb6822290b046f88c861159aa9421ec: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/15a5f02e61d24029b0a5757556085733
I20240503 08:00:52.792167 17108 catalog_manager.cc:4830] TS 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199): tablet 15a5f02e61d24029b0a5757556085733 (table rebalance_test_table_5 [id=d9c8ad70e1f0425d95eabb7b479ee693]) successfully deleted
I20240503 08:00:52.792361 17529 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
W20240503 08:00:52.793110 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08e9b83cd82f4d82ae552501be742896. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:52.795356 17926 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:52.811782 17397 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:52.849303 17397 consensus_queue.cc:237] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:52.851312 19635 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.851511 17926 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.851692 19504 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.851934 21133 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.852067 17791 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.852306 21133 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.852422 17529 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.852447 21900 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.852763 21900 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.852782 18057 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:52.853658 21133 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.853871 21847 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:52.856168 21844 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.856447 17926 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.857829 19635 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.858045 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d1de048a83e45548afac96309abf4d2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:52.858383 19504 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.859742 17529 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.860409 18056 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.861225 17121 catalog_manager.cc:5484] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 reported cstate change: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.861553 17791 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:52.864143 17246 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1de048a83e45548afac96309abf4d2 with delete_type TABLET_DATA_TOMBSTONED (TS 8eb6822290b046f88c861159aa9421ec not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44958
I20240503 08:00:52.864370 21995 tablet_replica.cc:330] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec: stopping tablet replica
I20240503 08:00:52.864495 21995 raft_consensus.cc:2229] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:52.864711 21995 raft_consensus.cc:2258] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:52.865403 21995 ts_tablet_manager.cc:1875] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:52.867383 21995 ts_tablet_manager.cc:1888] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:52.867448 21995 log.cc:1196] T 3d1de048a83e45548afac96309abf4d2 P 8eb6822290b046f88c861159aa9421ec: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3d1de048a83e45548afac96309abf4d2
I20240503 08:00:52.868928 17108 catalog_manager.cc:4830] TS 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199): tablet 3d1de048a83e45548afac96309abf4d2 (table rebalance_test_table_4 [id=7d40f4bfc49940ad8e10d7aac57f142d]) successfully deleted
I20240503 08:00:53.247339 22116 ts_tablet_manager.cc:902] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:53.247826 22116 tablet_copy_client.cc:308] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33199
I20240503 08:00:53.248282 17286 tablet_copy_service.cc:140] P 8eb6822290b046f88c861159aa9421ec: Received BeginTabletCopySession request for tablet 8a03fdbe7f824674a351fd4598dc2c63 from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:37849)
I20240503 08:00:53.248394 17286 tablet_copy_service.cc:161] P 8eb6822290b046f88c861159aa9421ec: Beginning new tablet copy session on tablet 8a03fdbe7f824674a351fd4598dc2c63 from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:37849: session id = b65378a282b9443484843de0b8906df9-8a03fdbe7f824674a351fd4598dc2c63
I20240503 08:00:53.249195 17286 tablet_copy_source_session.cc:205] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:53.250025 22116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:53.251871 22116 tablet_copy_client.cc:783] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:53.252017 22116 tablet_copy_client.cc:647] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:53.253010 22116 tablet_copy_client.cc:523] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:53.254092 22116 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:53.273557 22116 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:53.273809 22116 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:53.273939 22116 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.004s
I20240503 08:00:53.274271 22116 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:53.274380 22116 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:53.274456 22116 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:53.274698 22116 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:53.275154 17286 tablet_copy_service.cc:342] P 8eb6822290b046f88c861159aa9421ec: Request end of tablet copy session b65378a282b9443484843de0b8906df9-8a03fdbe7f824674a351fd4598dc2c63 received from {username='slave'} at 127.0.99.136:37849
I20240503 08:00:53.275244 17286 tablet_copy_service.cc:433] P 8eb6822290b046f88c861159aa9421ec: ending tablet copy session b65378a282b9443484843de0b8906df9-8a03fdbe7f824674a351fd4598dc2c63 on tablet 8a03fdbe7f824674a351fd4598dc2c63 with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:53.334576 22132 ts_tablet_manager.cc:902] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer d502c5c7f12a4e39aa25cd57f92c403d (127.0.99.135:41359)
I20240503 08:00:53.335050 22132 tablet_copy_client.cc:308] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:41359
I20240503 08:00:53.338727 18078 tablet_copy_service.cc:140] P d502c5c7f12a4e39aa25cd57f92c403d: Received BeginTabletCopySession request for tablet 08e9b83cd82f4d82ae552501be742896 from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:60647)
I20240503 08:00:53.338817 18078 tablet_copy_service.cc:161] P d502c5c7f12a4e39aa25cd57f92c403d: Beginning new tablet copy session on tablet 08e9b83cd82f4d82ae552501be742896 from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:60647: session id = 0665f4f85b984a8bba213ad479a88bad-08e9b83cd82f4d82ae552501be742896
I20240503 08:00:53.339450 18078 tablet_copy_source_session.cc:205] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:53.340227 22132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e9b83cd82f4d82ae552501be742896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:53.342020 22132 tablet_copy_client.cc:783] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:53.342170 22132 tablet_copy_client.cc:647] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:53.343173 22132 tablet_copy_client.cc:523] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:53.344201 22132 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:53.363514 22132 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:53.363737 22132 tablet_bootstrap.cc:492] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:53.363875 22132 ts_tablet_manager.cc:1367] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20240503 08:00:53.364203 22132 raft_consensus.cc:348] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:53.364311 22132 raft_consensus.cc:729] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:53.364392 22132 consensus_queue.cc:260] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:53.364682 22132 ts_tablet_manager.cc:1398] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240503 08:00:53.365175 18078 tablet_copy_service.cc:342] P d502c5c7f12a4e39aa25cd57f92c403d: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-08e9b83cd82f4d82ae552501be742896 received from {username='slave'} at 127.0.99.137:60647
I20240503 08:00:53.365262 18078 tablet_copy_service.cc:433] P d502c5c7f12a4e39aa25cd57f92c403d: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-08e9b83cd82f4d82ae552501be742896 on tablet 08e9b83cd82f4d82ae552501be742896 with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:53.729151 19373 raft_consensus.cc:1203] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:53.783413 19636 raft_consensus.cc:1203] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 08:00:54.112452 22157 raft_consensus.cc:1050] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d: attempting to promote NON_VOTER 0665f4f85b984a8bba213ad479a88bad to VOTER
I20240503 08:00:54.113011 22157 consensus_queue.cc:237] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:54.115300 19373 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.115299 17661 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.115669 17397 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.115710 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.116075 22182 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.116247 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.117204 19636 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.117627 22182 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.117760 17926 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.118078 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.118505 17530 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.118857 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.120330 17265 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.120666 22181 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.121467 22157 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.121804 19636 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.123036 17530 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.124094 17926 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.124307 17120 catalog_manager.cc:5484] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 reported cstate change: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.124665 19373 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.126102 17661 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.127373 17397 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.131175 17265 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } attrs { replace: true } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.132721 18057 consensus_queue.cc:237] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } }
I20240503 08:00:54.134503 19373 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.134733 17661 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.134840 17924 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.135079 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.135272 17397 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.135264 17530 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.135639 19636 raft_consensus.cc:1261] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer d502c5c7f12a4e39aa25cd57f92c403d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.135630 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.136013 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.136176 22184 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.136286 20977 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.136523 22156 consensus_queue.cc:1035] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.138154 22156 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.138567 17926 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.140173 17109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08e9b83cd82f4d82ae552501be742896 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:54.140173 17530 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.140792 17397 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.140869 17661 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.141469 19636 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.142758 17117 catalog_manager.cc:5484] T 08e9b83cd82f4d82ae552501be742896 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.144639 19373 raft_consensus.cc:2932] T 08e9b83cd82f4d82ae552501be742896 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8eb6822290b046f88c861159aa9421ec (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } }
I20240503 08:00:54.145632 17246 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e9b83cd82f4d82ae552501be742896 with delete_type TABLET_DATA_TOMBSTONED (TS 8eb6822290b046f88c861159aa9421ec not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44958
I20240503 08:00:54.146381 22196 tablet_replica.cc:330] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: stopping tablet replica
I20240503 08:00:54.146541 22196 raft_consensus.cc:2229] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:54.146982 22196 raft_consensus.cc:2258] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:54.147840 22196 ts_tablet_manager.cc:1875] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:54.149746 22196 ts_tablet_manager.cc:1888] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:54.149825 22196 log.cc:1196] T 08e9b83cd82f4d82ae552501be742896 P 8eb6822290b046f88c861159aa9421ec: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/08e9b83cd82f4d82ae552501be742896
I20240503 08:00:54.150454 17108 catalog_manager.cc:4830] TS 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199): tablet 08e9b83cd82f4d82ae552501be742896 (table rebalance_test_table_0 [id=44b3678d9fc54d9a83ac890b9447cbb3]) successfully deleted
I20240503 08:00:54.179715 21431 raft_consensus.cc:1050] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: attempting to promote NON_VOTER b65378a282b9443484843de0b8906df9 to VOTER
I20240503 08:00:54.180207 21431 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:54.182348 19636 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.182406 17530 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.182693 18057 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.182754 19504 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.182736 17661 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.182812 17791 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.183189 22214 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.183352 17397 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.183482 22164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.184208 22164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.184351 22215 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.184516 22213 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.184983 21431 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.185195 22216 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.188315 22215 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.189045 18057 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.189623 19636 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.190958 19505 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.191112 17791 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.193091 17119 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 reported cstate change: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.197618 17397 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.199447 17661 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.201454 17530 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b65378a282b9443484843de0b8906df9 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.203985 17265 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } }
I20240503 08:00:54.205483 19636 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.205619 19505 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.205961 17530 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.206096 22163 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.206408 22163 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.206549 22215 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.206625 18057 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.206954 22216 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.207649 17661 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.207962 17791 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:54.208217 22216 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.208775 22216 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:54.210813 22163 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.211848 19505 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.212278 17661 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.213002 17108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a03fdbe7f824674a351fd4598dc2c63 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:00:54.213544 17791 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.213789 18057 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.213902 17530 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.214092 19636 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.217028 17119 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 9 to 10, VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } }
I20240503 08:00:54.219395 17377 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a03fdbe7f824674a351fd4598dc2c63 with delete_type TABLET_DATA_TOMBSTONED (TS f0a940e3a6e340a69ec7ac31d26c9298 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39992
I20240503 08:00:54.219866 22225 tablet_replica.cc:330] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: stopping tablet replica
I20240503 08:00:54.220021 22225 raft_consensus.cc:2229] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:00:54.220351 22225 raft_consensus.cc:2258] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:00:54.221711 22225 ts_tablet_manager.cc:1875] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:00:54.223433 22225 ts_tablet_manager.cc:1888] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 08:00:54.223491 22225 log.cc:1196] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Deleting WAL directory at /tmp/dist-test-taskD2FJ4a/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714723143728567-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a03fdbe7f824674a351fd4598dc2c63
I20240503 08:00:54.223927 17108 catalog_manager.cc:4830] TS f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225): tablet 8a03fdbe7f824674a351fd4598dc2c63 (table rebalance_test_table_1 [id=fec5c63d23b74acb996bb2bdca9c4ed8]) successfully deleted
I20240503 08:00:54.432637 19354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:54.433827 17640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.437166 17510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.438138 17246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.446316 17905 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.455317 19610 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:54.456238 17377 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:54.472952 18037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.485023 19485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:54.497014 17774 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.598253 19354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:54.605206 17905 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.608171 17640 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.616887 19610 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:54.617424 19485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:00:54.624114 17377 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:00:54.624351 17246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:00:54.649348 18037 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.658432 17510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:00:54.677210 17774 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:00:54.730906 17397 consensus_queue.cc:237] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:54.732651 17265 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } }
I20240503 08:00:54.733888 17924 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.733902 19636 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.734284 19504 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.734387 17791 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.734826 22276 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.735859 18056 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.736018 18057 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.736568 17530 raft_consensus.cc:1261] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.736946 17662 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.736984 22276 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.737663 17530 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:00:54.737821 17199 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225): Couldn't send request to peer f0a940e3a6e340a69ec7ac31d26c9298. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.737989 21900 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:00:54.738137 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d1de048a83e45548afac96309abf4d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.738279 21900 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.738564 21900 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.738672 19504 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.738873 21900 consensus_queue.cc:1035] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.738987 17791 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.739482 19636 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.739881 22164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.740231 22164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.740517 22164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.740777 22164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.741035 22164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.741348 22164 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.743160 21900 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.743544 19504 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.745813 17924 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.746210 17115 catalog_manager.cc:5484] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 10 to 11, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:54.747257 17530 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.750849 18056 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } } }
I20240503 08:00:54.752848 19635 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } } }
I20240503 08:00:54.753736 17121 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d reported cstate change: config changed from index 10 to 11, NON_VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } } }
I20240503 08:00:54.753808 17530 consensus_queue.cc:237] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } }
I20240503 08:00:54.755784 19373 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.756366 17794 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:54.756453 17463 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863): Couldn't send request to peer 0ce150b975ff4040bedde6319d5cf3d5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.756788 17924 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.756889 17397 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.757207 17662 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } } }
I20240503 08:00:54.757400 22333 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.757524 17265 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.757720 22333 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.757997 22333 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.758261 22333 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.758423 17794 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } } }
I20240503 08:00:54.759789 21898 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.754978 22355 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } } }
I20240503 08:00:54.763748 18056 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.764525 19504 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } } }
I20240503 08:00:54.764840 17530 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } } }
I20240503 08:00:54.766438 22376 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.768412 22376 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } } }
I20240503 08:00:54.768857 19373 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } } }
I20240503 08:00:54.769955 18056 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } } }
I20240503 08:00:54.773790 17791 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } } }
I20240503 08:00:54.774489 19636 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.776242 17924 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } } }
I20240503 08:00:54.779057 17122 catalog_manager.cc:5484] T 143109561ad549d69a59992c02fe23af P 0665f4f85b984a8bba213ad479a88bad reported cstate change: config changed from index 7 to 8, NON_VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } } }
I20240503 08:00:54.779318 17662 consensus_queue.cc:237] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:54.779871 17397 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } } }
I20240503 08:00:54.781230 17265 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } } }
I20240503 08:00:54.781726 17397 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.782107 22020 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.782269 17530 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.782555 17263 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.783200 18056 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.783382 19636 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.783674 17924 raft_consensus.cc:1261] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 0ce150b975ff4040bedde6319d5cf3d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:54.783891 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba0615c2fbc840c7bbdfda22caf45de2. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.784500 22020 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.784824 22020 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.786948 17793 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.787611 18057 raft_consensus.cc:2932] T 3d1de048a83e45548afac96309abf4d2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.790509 22020 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.790839 22020 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.791110 22020 consensus_queue.cc:1035] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.801862 22020 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.802137 17924 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } }
I20240503 08:00:54.803743 17265 consensus_queue.cc:237] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:54.804678 18056 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.805028 19373 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.806053 17398 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.806119 19504 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.806334 17397 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.806707 17662 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.806814 22164 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.806918 17529 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.806931 17661 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.807323 22166 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.807621 22164 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.807653 22166 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.807812 22355 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.808037 19373 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.808269 17397 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
W20240503 08:00:54.809042 17859 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647): Couldn't send request to peer 7af8f2a481294a2cb93e5761947978cc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.809372 21984 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.809482 18057 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.809653 21984 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:00:54.809716 17201 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Couldn't send request to peer 84305d92b082406380b107114e4cd61b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.810000 22355 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.810156 17926 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.811443 22355 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.811760 22355 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.812395 17528 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.812666 17924 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.813933 17265 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.814407 22166 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.814615 17124 catalog_manager.cc:5484] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New cstate: current_term: 1 leader_uuid: "0ce150b975ff4040bedde6319d5cf3d5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:54.816308 18056 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.816677 21984 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.824554 17263 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.825651 22164 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.828015 19374 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.828260 22415 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } } }
I20240503 08:00:54.829517 17116 catalog_manager.cc:5484] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:54.830003 17662 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.831604 17398 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.832178 19504 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } } }
I20240503 08:00:54.832609 17530 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.834918 17115 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 reported cstate change: config changed from index 10 to 11, NON_VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } } }
I20240503 08:00:54.835166 17926 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.836880 17398 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } } }
I20240503 08:00:54.839444 17265 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } } }
I20240503 08:00:54.841667 19636 raft_consensus.cc:2932] T ba0615c2fbc840c7bbdfda22caf45de2 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0665f4f85b984a8bba213ad479a88bad (127.0.99.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } } }
I20240503 08:00:54.846964 17398 consensus_queue.cc:237] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:54.846833 18057 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.848968 19504 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.849622 17926 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.849637 22352 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.849994 22352 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.850268 17793 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.850404 18057 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 08:00:54.850423 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8500b198108649a7bcb614c93cf53a3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.850863 22353 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.852221 22351 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.852672 19374 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.853106 22351 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.854493 17265 raft_consensus.cc:1261] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.854883 22351 consensus_queue.cc:1035] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.855733 18056 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } } }
I20240503 08:00:54.855894 19372 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } } }
I20240503 08:00:54.857409 22351 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.859514 19504 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.860225 17793 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.862043 17115 catalog_manager.cc:5484] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 10 to 11, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:54.862284 17662 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } } }
I20240503 08:00:54.867985 17265 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.870944 18057 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.874433 17926 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.876210 19374 raft_consensus.cc:2932] T 8500b198108649a7bcb614c93cf53a3a P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.884171 17924 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:54.886395 17265 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.886395 17530 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.886713 22413 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.886940 19636 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.887266 18057 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.887513 17263 consensus_queue.cc:237] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:54.887969 22413 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.888278 22413 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.889403 17661 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.889403 17662 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.889730 19504 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.890069 22358 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.890642 19374 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.890995 17530 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.890998 17398 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:54.891333 22413 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.891527 18057 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 08:00:54.891582 17201 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Couldn't send request to peer 84305d92b082406380b107114e4cd61b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:54.891806 17861 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Couldn't send request to peer 84305d92b082406380b107114e4cd61b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.891834 22356 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.891937 22357 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.892218 22360 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.892362 22166 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.892645 22166 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.893893 17924 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 08:00:54.894734 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00708b0b8f2f4366944b244a9ba8c29f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.894985 22356 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.895082 22413 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.895498 22356 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.895540 19636 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.895911 17397 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.898274 17528 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.899315 17661 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.899797 17123 catalog_manager.cc:5484] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 8 to 9, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.899998 17120 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:54.900777 17662 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.903163 17528 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.904204 22357 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:54.905139 18057 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.905314 17924 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.908921 18057 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.909029 19374 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b65378a282b9443484843de0b8906df9 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:54.911895 17265 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.916528 19504 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.919847 17398 consensus_queue.cc:237] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:54.921686 18057 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.921761 19374 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.921762 17924 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.921773 17662 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.922077 22353 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.922305 22352 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.922546 21900 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.922654 22354 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.923790 17528 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 08:00:54.924122 17333 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Couldn't send request to peer 84305d92b082406380b107114e4cd61b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.924165 21900 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.925423 17265 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.925856 22353 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.927197 21900 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.927901 17265 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.928097 17924 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.929736 19374 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.930603 17120 catalog_manager.cc:5484] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec reported cstate change: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.932219 17662 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.933818 17528 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.938604 18057 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.946550 17924 consensus_queue.cc:237] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
W20240503 08:00:54.948386 17861 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Couldn't send request to peer 84305d92b082406380b107114e4cd61b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.948524 18057 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.949033 22166 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.949227 17265 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.949244 17398 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.949554 17662 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.949875 17528 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.950477 19374 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:00:54.950593 22166 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.950728 21984 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.950872 22416 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.951190 21984 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.951444 22414 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:00:54.954793 21984 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.956871 17528 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.957458 17662 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.957691 17119 catalog_manager.cc:5484] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:54.959093 17398 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.960709 19374 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.963500 18057 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.966948 17265 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.970513 17528 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:54.972498 19636 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.972718 19374 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.972839 22377 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 08:00:54.973193 17465 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 84305d92b082406380b107114e4cd61b (127.0.99.133:38825): Couldn't send request to peer 84305d92b082406380b107114e4cd61b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:54.973673 17924 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.973673 17662 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.973999 18057 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.974094 22521 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.974402 17265 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 08:00:54.974915 21898 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.975153 22377 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.975298 22333 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.976329 21898 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 08:00:54.977910 21898 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.978355 19374 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0665f4f85b984a8bba213ad479a88bad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.979985 17924 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.980475 17265 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.981535 19636 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P b65378a282b9443484843de0b8906df9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.981899 18057 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:54.982316 17120 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:54.986177 17662 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 84305d92b082406380b107114e4cd61b (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } }
I20240503 08:00:55.173625 22584 ts_tablet_manager.cc:902] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Initiating tablet copy from peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
I20240503 08:00:55.174366 22584 tablet_copy_client.cc:308] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39647
I20240503 08:00:55.175019 17550 tablet_copy_service.cc:140] P 7af8f2a481294a2cb93e5761947978cc: Received BeginTabletCopySession request for tablet 143109561ad549d69a59992c02fe23af from peer 0ce150b975ff4040bedde6319d5cf3d5 ({username='slave'} at 127.0.99.132:52541)
I20240503 08:00:55.175128 17550 tablet_copy_service.cc:161] P 7af8f2a481294a2cb93e5761947978cc: Beginning new tablet copy session on tablet 143109561ad549d69a59992c02fe23af from peer 0ce150b975ff4040bedde6319d5cf3d5 at {username='slave'} at 127.0.99.132:52541: session id = 0ce150b975ff4040bedde6319d5cf3d5-143109561ad549d69a59992c02fe23af
I20240503 08:00:55.176015 17550 tablet_copy_source_session.cc:205] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:55.178674 22584 ts_tablet_manager.cc:1875] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:55.180621 22584 ts_tablet_manager.cc:1888] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:00:55.180802 22584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143109561ad549d69a59992c02fe23af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.181988 22584 tablet_copy_client.cc:783] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.182152 22584 tablet_copy_client.cc:647] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.183475 22584 tablet_copy_client.cc:523] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:55.184752 22584 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap starting.
I20240503 08:00:55.191928 22595 ts_tablet_manager.cc:902] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:55.192626 22595 tablet_copy_client.cc:308] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:55.193217 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet e57bf40798824baf8050988874664154 from peer 7af8f2a481294a2cb93e5761947978cc ({username='slave'} at 127.0.99.131:41613)
I20240503 08:00:55.193311 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet e57bf40798824baf8050988874664154 from peer 7af8f2a481294a2cb93e5761947978cc at {username='slave'} at 127.0.99.131:41613: session id = 7af8f2a481294a2cb93e5761947978cc-e57bf40798824baf8050988874664154
I20240503 08:00:55.194154 17946 tablet_copy_source_session.cc:205] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:55.195077 22595 ts_tablet_manager.cc:1875] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:55.197197 22595 ts_tablet_manager.cc:1888] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 08:00:55.197345 22595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57bf40798824baf8050988874664154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.200098 22595 tablet_copy_client.cc:783] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.200242 22595 tablet_copy_client.cc:647] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.201699 22595 tablet_copy_client.cc:523] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:55.203362 22595 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap starting.
I20240503 08:00:55.210764 22584 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.211009 22584 tablet_bootstrap.cc:492] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Bootstrap complete.
I20240503 08:00:55.211158 22584 ts_tablet_manager.cc:1367] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Time spent bootstrapping tablet: real 0.026s user 0.025s sys 0.000s
I20240503 08:00:55.211539 22584 raft_consensus.cc:348] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } }
I20240503 08:00:55.211663 22584 raft_consensus.cc:729] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ce150b975ff4040bedde6319d5cf3d5, State: Initialized, Role: LEARNER
I20240503 08:00:55.211753 22584 consensus_queue.cc:260] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: true } }
I20240503 08:00:55.212400 22584 ts_tablet_manager.cc:1398] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 08:00:55.212872 17550 tablet_copy_service.cc:342] P 7af8f2a481294a2cb93e5761947978cc: Request end of tablet copy session 0ce150b975ff4040bedde6319d5cf3d5-143109561ad549d69a59992c02fe23af received from {username='slave'} at 127.0.99.132:52541
I20240503 08:00:55.212936 17550 tablet_copy_service.cc:433] P 7af8f2a481294a2cb93e5761947978cc: ending tablet copy session 0ce150b975ff4040bedde6319d5cf3d5-143109561ad549d69a59992c02fe23af on tablet 143109561ad549d69a59992c02fe23af with peer 0ce150b975ff4040bedde6319d5cf3d5
I20240503 08:00:55.227371 22595 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.227629 22595 tablet_bootstrap.cc:492] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Bootstrap complete.
I20240503 08:00:55.227788 22595 ts_tablet_manager.cc:1367] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20240503 08:00:55.228475 22595 raft_consensus.cc:348] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } }
I20240503 08:00:55.228626 22595 raft_consensus.cc:729] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7af8f2a481294a2cb93e5761947978cc, State: Initialized, Role: LEARNER
I20240503 08:00:55.228726 22595 consensus_queue.cc:260] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: true } }
I20240503 08:00:55.229045 22595 ts_tablet_manager.cc:1398] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240503 08:00:55.229637 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session 7af8f2a481294a2cb93e5761947978cc-e57bf40798824baf8050988874664154 received from {username='slave'} at 127.0.99.131:41613
I20240503 08:00:55.229729 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session 7af8f2a481294a2cb93e5761947978cc-e57bf40798824baf8050988874664154 on tablet e57bf40798824baf8050988874664154 with peer 7af8f2a481294a2cb93e5761947978cc
I20240503 08:00:55.258044 22631 ts_tablet_manager.cc:902] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Initiating tablet copy from peer 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:55.258770 22631 tablet_copy_client.cc:308] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33199
I20240503 08:00:55.259475 17286 tablet_copy_service.cc:140] P 8eb6822290b046f88c861159aa9421ec: Received BeginTabletCopySession request for tablet 8a03fdbe7f824674a351fd4598dc2c63 from peer f0a940e3a6e340a69ec7ac31d26c9298 ({username='slave'} at 127.0.99.130:48861)
I20240503 08:00:55.259572 17286 tablet_copy_service.cc:161] P 8eb6822290b046f88c861159aa9421ec: Beginning new tablet copy session on tablet 8a03fdbe7f824674a351fd4598dc2c63 from peer f0a940e3a6e340a69ec7ac31d26c9298 at {username='slave'} at 127.0.99.130:48861: session id = f0a940e3a6e340a69ec7ac31d26c9298-8a03fdbe7f824674a351fd4598dc2c63
I20240503 08:00:55.260442 17286 tablet_copy_source_session.cc:205] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:55.261488 22631 ts_tablet_manager.cc:1875] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:55.263311 22631 ts_tablet_manager.cc:1888] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240503 08:00:55.263463 22631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a03fdbe7f824674a351fd4598dc2c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.264401 22631 tablet_copy_client.cc:783] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.264533 22631 tablet_copy_client.cc:647] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.265867 22631 tablet_copy_client.cc:523] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:55.267040 22631 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap starting.
I20240503 08:00:55.287662 22631 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.287923 22631 tablet_bootstrap.cc:492] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Bootstrap complete.
I20240503 08:00:55.288105 22631 ts_tablet_manager.cc:1367] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240503 08:00:55.288516 22631 raft_consensus.cc:348] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } }
I20240503 08:00:55.288663 22631 raft_consensus.cc:729] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0a940e3a6e340a69ec7ac31d26c9298, State: Initialized, Role: LEARNER
I20240503 08:00:55.288763 22631 consensus_queue.cc:260] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: true } }
I20240503 08:00:55.289083 22631 ts_tablet_manager.cc:1398] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:55.289640 17286 tablet_copy_service.cc:342] P 8eb6822290b046f88c861159aa9421ec: Request end of tablet copy session f0a940e3a6e340a69ec7ac31d26c9298-8a03fdbe7f824674a351fd4598dc2c63 received from {username='slave'} at 127.0.99.130:48861
I20240503 08:00:55.289724 17286 tablet_copy_service.cc:433] P 8eb6822290b046f88c861159aa9421ec: ending tablet copy session f0a940e3a6e340a69ec7ac31d26c9298-8a03fdbe7f824674a351fd4598dc2c63 on tablet 8a03fdbe7f824674a351fd4598dc2c63 with peer f0a940e3a6e340a69ec7ac31d26c9298
I20240503 08:00:55.297132 22637 ts_tablet_manager.cc:902] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132:34863)
I20240503 08:00:55.297554 22637 tablet_copy_client.cc:308] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:34863
I20240503 08:00:55.297986 17682 tablet_copy_service.cc:140] P 0ce150b975ff4040bedde6319d5cf3d5: Received BeginTabletCopySession request for tablet ba0615c2fbc840c7bbdfda22caf45de2 from peer 0665f4f85b984a8bba213ad479a88bad ({username='slave'} at 127.0.99.137:55309)
I20240503 08:00:55.298089 17682 tablet_copy_service.cc:161] P 0ce150b975ff4040bedde6319d5cf3d5: Beginning new tablet copy session on tablet ba0615c2fbc840c7bbdfda22caf45de2 from peer 0665f4f85b984a8bba213ad479a88bad at {username='slave'} at 127.0.99.137:55309: session id = 0665f4f85b984a8bba213ad479a88bad-ba0615c2fbc840c7bbdfda22caf45de2
I20240503 08:00:55.298713 17682 tablet_copy_source_session.cc:205] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:55.299458 22637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0615c2fbc840c7bbdfda22caf45de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.301441 22637 tablet_copy_client.cc:783] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.301585 22637 tablet_copy_client.cc:647] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.302592 22637 tablet_copy_client.cc:523] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:55.303514 22637 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap starting.
I20240503 08:00:55.311820 22643 ts_tablet_manager.cc:902] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:55.312320 22643 tablet_copy_client.cc:308] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:55.312760 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet 8500b198108649a7bcb614c93cf53a3a from peer b65378a282b9443484843de0b8906df9 ({username='slave'} at 127.0.99.136:39241)
I20240503 08:00:55.312851 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet 8500b198108649a7bcb614c93cf53a3a from peer b65378a282b9443484843de0b8906df9 at {username='slave'} at 127.0.99.136:39241: session id = b65378a282b9443484843de0b8906df9-8500b198108649a7bcb614c93cf53a3a
I20240503 08:00:55.313606 17418 tablet_copy_source_session.cc:205] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:55.314350 22643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500b198108649a7bcb614c93cf53a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.315955 22643 tablet_copy_client.cc:783] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.316087 22643 tablet_copy_client.cc:647] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.317090 22643 tablet_copy_client.cc:523] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:55.317962 22643 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: Bootstrap starting.
I20240503 08:00:55.324117 22637 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.324390 22637 tablet_bootstrap.cc:492] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: Bootstrap complete.
I20240503 08:00:55.324551 22637 ts_tablet_manager.cc:1367] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.002s
I20240503 08:00:55.325048 22637 raft_consensus.cc:348] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:55.325197 22637 raft_consensus.cc:729] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0665f4f85b984a8bba213ad479a88bad, State: Initialized, Role: LEARNER
I20240503 08:00:55.325297 22637 consensus_queue.cc:260] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: NON_VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: true } }
I20240503 08:00:55.325601 22637 ts_tablet_manager.cc:1398] T ba0615c2fbc840c7bbdfda22caf45de2 P 0665f4f85b984a8bba213ad479a88bad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:55.326239 17682 tablet_copy_service.cc:342] P 0ce150b975ff4040bedde6319d5cf3d5: Request end of tablet copy session 0665f4f85b984a8bba213ad479a88bad-ba0615c2fbc840c7bbdfda22caf45de2 received from {username='slave'} at 127.0.99.137:55309
I20240503 08:00:55.326318 17682 tablet_copy_service.cc:433] P 0ce150b975ff4040bedde6319d5cf3d5: ending tablet copy session 0665f4f85b984a8bba213ad479a88bad-ba0615c2fbc840c7bbdfda22caf45de2 on tablet ba0615c2fbc840c7bbdfda22caf45de2 with peer 0665f4f85b984a8bba213ad479a88bad
I20240503 08:00:55.342056 22643 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.342331 22643 tablet_bootstrap.cc:492] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: Bootstrap complete.
I20240503 08:00:55.342509 22643 ts_tablet_manager.cc:1367] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240503 08:00:55.342890 22643 raft_consensus.cc:348] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:55.343005 22643 raft_consensus.cc:729] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b65378a282b9443484843de0b8906df9, State: Initialized, Role: LEARNER
I20240503 08:00:55.343120 22643 consensus_queue.cc:260] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:55.343359 22643 ts_tablet_manager.cc:1398] T 8500b198108649a7bcb614c93cf53a3a P b65378a282b9443484843de0b8906df9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:55.343835 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session b65378a282b9443484843de0b8906df9-8500b198108649a7bcb614c93cf53a3a received from {username='slave'} at 127.0.99.136:39241
I20240503 08:00:55.343909 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session b65378a282b9443484843de0b8906df9-8500b198108649a7bcb614c93cf53a3a on tablet 8500b198108649a7bcb614c93cf53a3a with peer b65378a282b9443484843de0b8906df9
I20240503 08:00:55.351863 22637 ts_tablet_manager.cc:902] T 3d1de048a83e45548afac96309abf4d2 P 0665f4f85b984a8bba213ad479a88bad: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
F20240503 08:00:55.352025 22637 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-00708b0b8f2f4366944b244a9ba8c29f.queue_time_us, tablet-download-pool-00708b0b8f2f4366944b244a9ba8c29f.queue_time_us) (tablet-download-pool-00708b0b8f2f4366944b244a9ba8c29f.run_cpu_time_us, tablet-download-pool-00708b0b8f2f4366944b244a9ba8c29f.run_cpu_time_us) (tablet-download-pool-00708b0b8f2f4366944b244a9ba8c29f.run_wall_time_us, tablet-download-pool-00708b0b8f2f4366944b244a9ba8c29f.run_wall_time_us) (tablet-download-pool-08e9b83cd82f4d82ae552501be742896.queue_time_us, tablet-download-pool-08e9b83cd82f4d82ae552501be742896.queue_time_us) (tablet-download-pool-08e9b83cd82f4d82ae552501be742896.run_cpu_time_us, tablet-download-pool-08e9b83cd82f4d82ae552501be742896.run_cpu_time_us) (tablet-download-pool-08e9b83cd82f4d82ae552501be742896.run_wall_time_us, tablet-download-pool-08e9b83cd82f4d82ae552501be742896.run_wall_time_us) (tablet-download-pool-10e598a3a124472eab35c97689057ace.queue_time_us, tablet-download-pool-10e598a3a124472eab35c97689057ace.queue_time_us) (tablet-download-pool-10e598a3a124472eab35c97689057ace.run_cpu_time_us, tablet-download-pool-10e598a3a124472eab35c97689057ace.run_cpu_time_us) (tablet-download-pool-10e598a3a124472eab35c97689057ace.run_wall_time_us, tablet-download-pool-10e598a3a124472eab35c97689057ace.run_wall_time_us) (tablet-download-pool-143109561ad549d69a59992c02fe23af.queue_time_us, tablet-download-pool-143109561ad549d69a59992c02fe23af.queue_time_us) (tablet-download-pool-143109561ad549d69a59992c02fe23af.run_cpu_time_us, tablet-download-pool-143109561ad549d69a59992c02fe23af.run_cpu_time_us) (tablet-download-pool-143109561ad549d69a59992c02fe23af.run_wall_time_us, tablet-download-pool-143109561ad549d69a59992c02fe23af.run_wall_time_us) (tablet-download-pool-15a5f02e61d24029b0a5757556085733.queue_time_us, tablet-download-pool-15a5f02e61d24029b0a5757556085733.queue_time_us) (tablet-download-pool-15a5f02e61d24029b0a5757556085733.run_cpu_time_us, tablet-download-pool-15a5f02e61d24029b0a5757556085733.run_cpu_time_us) (tablet-download-pool-15a5f02e61d24029b0a5757556085733.run_wall_time_us, tablet-download-pool-15a5f02e61d24029b0a5757556085733.run_wall_time_us) (tablet-download-pool-304ad5f1abf44548a72961d3f655cf61.queue_time_us, tablet-download-pool-304ad5f1abf44548a72961d3f655cf61.queue_time_us) (tablet-download-pool-304ad5f1abf44548a72961d3f655cf61.run_cpu_time_us, tablet-download-pool-304ad5f1abf44548a72961d3f655cf61.run_cpu_time_us) (tablet-download-pool-304ad5f1abf44548a72961d3f655cf61.run_wall_time_us, tablet-download-pool-304ad5f1abf44548a72961d3f655cf61.run_wall_time_us) (tablet-download-pool-3d1de048a83e45548afac96309abf4d2.queue_time_us, tablet-download-pool-3d1de048a83e45548afac96309abf4d2.queue_time_us) (tablet-download-pool-5154e7f64d1542979f9363a429fa469c.queue_time_us, tablet-download-pool-5154e7f64d1542979f9363a429fa469c.queue_time_us) (tablet-download-pool-5154e7f64d1542979f9363a429fa469c.run_cpu_time_us, tablet-download-pool-5154e7f64d1542979f9363a429fa469c.run_cpu_time_us) (tablet-download-pool-5154e7f64d1542979f9363a429fa469c.run_wall_time_us, tablet-download-pool-5154e7f64d1542979f9363a429fa469c.run_wall_time_us) (tablet-download-pool-70f073cc41d54638a2c51746b94a5ece.queue_time_us, tablet-download-pool-70f073cc41d54638a2c51746b94a5ece.queue_time_us) (tablet-download-pool-70f073cc41d54638a2c51746b94a5ece.run_cpu_time_us, tablet-download-pool-70f073cc41d54638a2c51746b94a5ece.run_cpu_time_us) (tablet-download-pool-70f073cc41d54638a2c51746b94a5ece.run_wall_time_us, tablet-download-pool-70f073cc41d54638a2c51746b94a5ece.run_wall_time_us) (tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.queue_time_us, tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.queue_time_us) (tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.run_cpu_time_us, tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.run_cpu_time_us) (tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.run_wall_time_us, tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.run_wall_time_us) (tablet-download-pool-882c64fe082243fbbb3099f2bf6e33e6.queue_time_us, tablet-download-pool-882c64fe082243fbbb3099f2bf6e33e6.queue_time_us) (tablet-download-pool-882c64fe082243fbbb3099f2bf6e33e6.run_cpu_time_us, tablet-download-pool-882c64fe082243fbbb3099f2bf6e33e6.run_cpu_time_us) (tablet-download-pool-882c64fe082243fbbb3099f2bf6e33e6.run_wall_time_us, tablet-download-pool-882c64fe082243fbbb3099f2bf6e33e6.run_wall_time_us) (tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.queue_time_us, tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.queue_time_us) (tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.run_cpu_time_us, tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.run_cpu_time_us) (tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.run_wall_time_us, tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.run_wall_time_us) (tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.queue_time_us, tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.queue_time_us) (tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.run_cpu_time_us, tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.run_cpu_time_us) (tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.run_wall_time_us, tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.run_wall_time_us) (tablet-download-pool-e57bf40798824baf8050988874664154.queue_time_us, tablet-download-pool-e57bf40798824baf8050988874664154.queue_time_us) (tablet-download-pool-e57bf40798824baf8050988874664154.run_cpu_time_us, tablet-download-pool-e57bf40798824baf8050988874664154.run_cpu_time_us) (tablet-download-pool-e57bf40798824baf8050988874664154.run_wall_time_us, tablet-download-pool-e57bf40798824baf8050988874664154.run_wall_time_us) (tablet-download-pool-ed32a1099e044448aaa0c9c9f58f74e6.queue_time_us, tablet-download-pool-ed32a1099e044448aaa0c9c9f58f74e6.queue_time_us) (tablet-download-pool-ed32a1099e044448aaa0c9c9f58f74e6.run_cpu_time_us, tablet-download-pool-ed32a1099e044448aaa0c9c9f58f74e6.run_cpu_time_us) (tablet-download-pool-ed32a1099e044448aaa0c9c9f58f74e6.run_wall_time_us, tablet-download-pool-ed32a1099e044448aaa0c9c9f58f74e6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f880c7b6dcd google::LogMessage::Fail() at ??:0
@ 0x7f880c7bab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f880c7b67cc google::LogMessage::Flush() at ??:0
@ 0x7f880c7b7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f880dc20044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f880dc04f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f880dc02efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f88107b577a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f88107bea45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f881083f76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f88108534f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f881083e226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f881084dc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x560be997475c std::function<>::operator()() at ??:0
@ 0x7f880dc07e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f880dc086ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f880dc0a03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x560be997475c std::function<>::operator()() at ??:0
@ 0x7f880dbf7807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f880e4356db start_thread at ??:0
I20240503 08:00:55.391173 22673 ts_tablet_manager.cc:902] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Initiating tablet copy from peer 7af8f2a481294a2cb93e5761947978cc (127.0.99.131:39647)
@ 0x7f880b96d71f clone at ??:0
I20240503 08:00:55.392477 22673 tablet_copy_client.cc:308] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:39647
I20240503 08:00:55.393349 17550 tablet_copy_service.cc:140] P 7af8f2a481294a2cb93e5761947978cc: Received BeginTabletCopySession request for tablet 10e598a3a124472eab35c97689057ace from peer 84305d92b082406380b107114e4cd61b ({username='slave'} at 127.0.99.133:36527)
I20240503 08:00:55.393450 17550 tablet_copy_service.cc:161] P 7af8f2a481294a2cb93e5761947978cc: Beginning new tablet copy session on tablet 10e598a3a124472eab35c97689057ace from peer 84305d92b082406380b107114e4cd61b at {username='slave'} at 127.0.99.133:36527: session id = 84305d92b082406380b107114e4cd61b-10e598a3a124472eab35c97689057ace
I20240503 08:00:55.394361 17550 tablet_copy_source_session.cc:205] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:55.395092 22679 ts_tablet_manager.cc:902] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Initiating tablet copy from peer f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130:43225)
I20240503 08:00:55.395570 22679 tablet_copy_client.cc:308] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43225
I20240503 08:00:55.395718 22673 ts_tablet_manager.cc:1875] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:55.397951 22673 ts_tablet_manager.cc:1888] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:00:55.398106 22673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e598a3a124472eab35c97689057ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.398324 17418 tablet_copy_service.cc:140] P f0a940e3a6e340a69ec7ac31d26c9298: Received BeginTabletCopySession request for tablet 304ad5f1abf44548a72961d3f655cf61 from peer 84305d92b082406380b107114e4cd61b ({username='slave'} at 127.0.99.133:40529)
I20240503 08:00:55.398408 17418 tablet_copy_service.cc:161] P f0a940e3a6e340a69ec7ac31d26c9298: Beginning new tablet copy session on tablet 304ad5f1abf44548a72961d3f655cf61 from peer 84305d92b082406380b107114e4cd61b at {username='slave'} at 127.0.99.133:40529: session id = 84305d92b082406380b107114e4cd61b-304ad5f1abf44548a72961d3f655cf61
I20240503 08:00:55.399263 17418 tablet_copy_source_session.cc:205] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:55.400868 22673 tablet_copy_client.cc:783] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.401026 22673 tablet_copy_client.cc:647] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.402398 22673 tablet_copy_client.cc:523] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:55.403363 22679 ts_tablet_manager.cc:1875] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:55.404067 22673 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:55.405153 22679 ts_tablet_manager.cc:1888] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:00:55.405290 22679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304ad5f1abf44548a72961d3f655cf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.406265 22679 tablet_copy_client.cc:783] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.406366 22679 tablet_copy_client.cc:647] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.409221 22679 tablet_copy_client.cc:523] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:55.410400 22679 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:55.424140 22697 ts_tablet_manager.cc:902] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:55.424672 22697 tablet_copy_client.cc:308] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:55.425737 22643 ts_tablet_manager.cc:902] T 00708b0b8f2f4366944b244a9ba8c29f P b65378a282b9443484843de0b8906df9: Initiating tablet copy from peer 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
F20240503 08:00:55.425900 22643 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-00708b0b8f2f4366944b244a9ba8c29f.queue_time_us, tablet-download-pool-00708b0b8f2f4366944b244a9ba8c29f.queue_time_us) (tablet-download-pool-08e9b83cd82f4d82ae552501be742896.queue_time_us, tablet-download-pool-08e9b83cd82f4d82ae552501be742896.queue_time_us) (tablet-download-pool-08e9b83cd82f4d82ae552501be742896.run_cpu_time_us, tablet-download-pool-08e9b83cd82f4d82ae552501be742896.run_cpu_time_us) (tablet-download-pool-08e9b83cd82f4d82ae552501be742896.run_wall_time_us, tablet-download-pool-08e9b83cd82f4d82ae552501be742896.run_wall_time_us) (tablet-download-pool-10e598a3a124472eab35c97689057ace.queue_time_us, tablet-download-pool-10e598a3a124472eab35c97689057ace.queue_time_us) (tablet-download-pool-10e598a3a124472eab35c97689057ace.run_cpu_time_us, tablet-download-pool-10e598a3a124472eab35c97689057ace.run_cpu_time_us) (tablet-download-pool-10e598a3a124472eab35c97689057ace.run_wall_time_us, tablet-download-pool-10e598a3a124472eab35c97689057ace.run_wall_time_us) (tablet-download-pool-15a5f02e61d24029b0a5757556085733.queue_time_us, tablet-download-pool-15a5f02e61d24029b0a5757556085733.queue_time_us) (tablet-download-pool-15a5f02e61d24029b0a5757556085733.run_cpu_time_us, tablet-download-pool-15a5f02e61d24029b0a5757556085733.run_cpu_time_us) (tablet-download-pool-15a5f02e61d24029b0a5757556085733.run_wall_time_us, tablet-download-pool-15a5f02e61d24029b0a5757556085733.run_wall_time_us) (tablet-download-pool-3d1de048a83e45548afac96309abf4d2.queue_time_us, tablet-download-pool-3d1de048a83e45548afac96309abf4d2.queue_time_us) (tablet-download-pool-3d1de048a83e45548afac96309abf4d2.run_cpu_time_us, tablet-download-pool-3d1de048a83e45548afac96309abf4d2.run_cpu_time_us) (tablet-download-pool-3d1de048a83e45548afac96309abf4d2.run_wall_time_us, tablet-download-pool-3d1de048a83e45548afac96309abf4d2.run_wall_time_us) (tablet-download-pool-42be6135d7654954b6886c32dfc688c5.queue_time_us, tablet-download-pool-42be6135d7654954b6886c32dfc688c5.queue_time_us) (tablet-download-pool-42be6135d7654954b6886c32dfc688c5.run_cpu_time_us, tablet-download-pool-42be6135d7654954b6886c32dfc688c5.run_cpu_time_us) (tablet-download-pool-42be6135d7654954b6886c32dfc688c5.run_wall_time_us, tablet-download-pool-42be6135d7654954b6886c32dfc688c5.run_wall_time_us) (tablet-download-pool-5154e7f64d1542979f9363a429fa469c.queue_time_us, tablet-download-pool-5154e7f64d1542979f9363a429fa469c.queue_time_us) (tablet-download-pool-5154e7f64d1542979f9363a429fa469c.run_cpu_time_us, tablet-download-pool-5154e7f64d1542979f9363a429fa469c.run_cpu_time_us) (tablet-download-pool-5154e7f64d1542979f9363a429fa469c.run_wall_time_us, tablet-download-pool-5154e7f64d1542979f9363a429fa469c.run_wall_time_us) (tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.queue_time_us, tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.queue_time_us) (tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.run_cpu_time_us, tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.run_cpu_time_us) (tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.run_wall_time_us, tablet-download-pool-8500b198108649a7bcb614c93cf53a3a.run_wall_time_us) (tablet-download-pool-8a03fdbe7f824674a351fd4598dc2c63.queue_time_us, tablet-download-pool-8a03fdbe7f824674a351fd4598dc2c63.queue_time_us) (tablet-download-pool-8a03fdbe7f824674a351fd4598dc2c63.run_cpu_time_us, tablet-download-pool-8a03fdbe7f824674a351fd4598dc2c63.run_cpu_time_us) (tablet-download-pool-8a03fdbe7f824674a351fd4598dc2c63.run_wall_time_us, tablet-download-pool-8a03fdbe7f824674a351fd4598dc2c63.run_wall_time_us) (tablet-download-pool-99d8d89b8561420781b7e7a9b9c778a0.queue_time_us, tablet-download-pool-99d8d89b8561420781b7e7a9b9c778a0.queue_time_us) (tablet-download-pool-99d8d89b8561420781b7e7a9b9c778a0.run_cpu_time_us, tablet-download-pool-99d8d89b8561420781b7e7a9b9c778a0.run_cpu_time_us) (tablet-download-pool-99d8d89b8561420781b7e7a9b9c778a0.run_wall_time_us, tablet-download-pool-99d8d89b8561420781b7e7a9b9c778a0.run_wall_time_us) (tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.queue_time_us, tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.queue_time_us) (tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.run_cpu_time_us, tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.run_cpu_time_us) (tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.run_wall_time_us, tablet-download-pool-a79ddd7d381c4ccbbd1a1e7d3ec8366f.run_wall_time_us) (tablet-download-pool-ab988cc96fc940239e18f54ec9c9b401.queue_time_us, tablet-download-pool-ab988cc96fc940239e18f54ec9c9b401.queue_time_us) (tablet-download-pool-ab988cc96fc940239e18f54ec9c9b401.run_cpu_time_us, tablet-download-pool-ab988cc96fc940239e18f54ec9c9b401.run_cpu_time_us) (tablet-download-pool-ab988cc96fc940239e18f54ec9c9b401.run_wall_time_us, tablet-download-pool-ab988cc96fc940239e18f54ec9c9b401.run_wall_time_us) (tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.queue_time_us, tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.queue_time_us) (tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.run_cpu_time_us, tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.run_cpu_time_us) (tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.run_wall_time_us, tablet-download-pool-ba0615c2fbc840c7bbdfda22caf45de2.run_wall_time_us) (tablet-download-pool-bbe0edacc8ce4ba8be5e533c82ea92e0.queue_time_us, tablet-download-pool-bbe0edacc8ce4ba8be5e533c82ea92e0.queue_time_us) (tablet-download-pool-bbe0edacc8ce4ba8be5e533c82ea92e0.run_cpu_time_us, tablet-download-pool-bbe0edacc8ce4ba8be5e533c82ea92e0.run_cpu_time_us) (tablet-download-pool-bbe0edacc8ce4ba8be5e533c82ea92e0.run_wall_time_us, tablet-download-pool-bbe0edacc8ce4ba8be5e533c82ea92e0.run_wall_time_us) (tablet-download-pool-ecc1f8b69268429f89cffb380e45e020.queue_time_us, tablet-download-pool-ecc1f8b69268429f89cffb380e45e020.queue_time_us) (tablet-download-pool-ecc1f8b69268429f89cffb380e45e020.run_cpu_time_us, tablet-download-pool-ecc1f8b69268429f89cffb380e45e020.run_cpu_time_us) (tablet-download-pool-ecc1f8b69268429f89cffb380e45e020.run_wall_time_us, tablet-download-pool-ecc1f8b69268429f89cffb380e45e020.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f73bd2d2dcd google::LogMessage::Fail() at ??:0
@ 0x7f73bd2d6b93 google::LogMessage::SendToLog() at ??:0
I20240503 08:00:55.433176 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet 70f073cc41d54638a2c51746b94a5ece from peer 84305d92b082406380b107114e4cd61b ({username='slave'} at 127.0.99.133:48107)
I20240503 08:00:55.433288 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet 70f073cc41d54638a2c51746b94a5ece from peer 84305d92b082406380b107114e4cd61b at {username='slave'} at 127.0.99.133:48107: session id = 84305d92b082406380b107114e4cd61b-70f073cc41d54638a2c51746b94a5ece
@ 0x7f73bd2d27cc google::LogMessage::Flush() at ??:0
I20240503 08:00:55.434264 17946 tablet_copy_source_session.cc:205] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:55.434518 22673 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.435022 22697 ts_tablet_manager.cc:1875] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f73bd2d3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 08:00:55.437274 22697 ts_tablet_manager.cc:1888] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:00:55.437424 22697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f073cc41d54638a2c51746b94a5ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.438685 22697 tablet_copy_client.cc:783] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.438804 22697 tablet_copy_client.cc:647] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.439378 22673 tablet_bootstrap.cc:492] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Bootstrap complete.
I20240503 08:00:55.439543 22673 ts_tablet_manager.cc:1367] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.036s user 0.019s sys 0.009s
I20240503 08:00:55.439970 22697 tablet_copy_client.cc:523] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f73be73c044 kudu::TraceMetrics::InternName() at ??:0
I20240503 08:00:55.441099 22697 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:55.444583 22711 ts_tablet_manager.cc:902] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Initiating tablet copy from peer 8eb6822290b046f88c861159aa9421ec (127.0.99.129:33199)
I20240503 08:00:55.445171 22711 tablet_copy_client.cc:308] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33199
I20240503 08:00:55.445432 22673 raft_consensus.cc:348] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:55.445559 22673 raft_consensus.cc:729] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: LEARNER
I20240503 08:00:55.445595 17286 tablet_copy_service.cc:140] P 8eb6822290b046f88c861159aa9421ec: Received BeginTabletCopySession request for tablet 00708b0b8f2f4366944b244a9ba8c29f from peer 84305d92b082406380b107114e4cd61b ({username='slave'} at 127.0.99.133:56087)
I20240503 08:00:55.445698 17286 tablet_copy_service.cc:161] P 8eb6822290b046f88c861159aa9421ec: Beginning new tablet copy session on tablet 00708b0b8f2f4366944b244a9ba8c29f from peer 84305d92b082406380b107114e4cd61b at {username='slave'} at 127.0.99.133:56087: session id = 84305d92b082406380b107114e4cd61b-00708b0b8f2f4366944b244a9ba8c29f
I20240503 08:00:55.445662 22673 consensus_queue.cc:260] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:55.446465 22673 ts_tablet_manager.cc:1398] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 08:00:55.446555 17286 tablet_copy_source_session.cc:205] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:00:55.446934 17550 tablet_copy_service.cc:342] P 7af8f2a481294a2cb93e5761947978cc: Request end of tablet copy session 84305d92b082406380b107114e4cd61b-10e598a3a124472eab35c97689057ace received from {username='slave'} at 127.0.99.133:36527
I20240503 08:00:55.447005 17550 tablet_copy_service.cc:433] P 7af8f2a481294a2cb93e5761947978cc: ending tablet copy session 84305d92b082406380b107114e4cd61b-10e598a3a124472eab35c97689057ace on tablet 10e598a3a124472eab35c97689057ace with peer 84305d92b082406380b107114e4cd61b
@ 0x7f73be720f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f73be71eefc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 08:00:55.450947 22711 ts_tablet_manager.cc:1875] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:00:55.452819 22711 ts_tablet_manager.cc:1888] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:00:55.452957 22711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00708b0b8f2f4366944b244a9ba8c29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.454780 22711 tablet_copy_client.cc:783] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.454903 22711 tablet_copy_client.cc:647] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.456225 22711 tablet_copy_client.cc:523] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f73c12d177a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 08:00:55.457675 22711 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
@ 0x7f73c12daa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f73c135b76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:00:55.472328 22697 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.472594 22697 tablet_bootstrap.cc:492] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Bootstrap complete.
I20240503 08:00:55.472754 22697 ts_tablet_manager.cc:1367] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.000s
I20240503 08:00:55.473192 22697 raft_consensus.cc:348] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:55.473325 22697 raft_consensus.cc:729] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: LEARNER
I20240503 08:00:55.473423 22697 consensus_queue.cc:260] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:55.474197 22697 ts_tablet_manager.cc:1398] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 08:00:55.475364 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session 84305d92b082406380b107114e4cd61b-70f073cc41d54638a2c51746b94a5ece received from {username='slave'} at 127.0.99.133:48107
I20240503 08:00:55.475445 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session 84305d92b082406380b107114e4cd61b-70f073cc41d54638a2c51746b94a5ece on tablet 70f073cc41d54638a2c51746b94a5ece with peer 84305d92b082406380b107114e4cd61b
I20240503 08:00:55.476045 22679 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.476294 22679 tablet_bootstrap.cc:492] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Bootstrap complete.
I20240503 08:00:55.476433 22679 ts_tablet_manager.cc:1367] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.066s user 0.021s sys 0.003s
I20240503 08:00:55.476768 22679 raft_consensus.cc:348] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:55.476874 22679 raft_consensus.cc:729] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: LEARNER
I20240503 08:00:55.476958 22679 consensus_queue.cc:260] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:55.477669 22679 ts_tablet_manager.cc:1398] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 08:00:55.478300 17418 tablet_copy_service.cc:342] P f0a940e3a6e340a69ec7ac31d26c9298: Request end of tablet copy session 84305d92b082406380b107114e4cd61b-304ad5f1abf44548a72961d3f655cf61 received from {username='slave'} at 127.0.99.133:40529
I20240503 08:00:55.478374 17418 tablet_copy_service.cc:433] P f0a940e3a6e340a69ec7ac31d26c9298: ending tablet copy session 84305d92b082406380b107114e4cd61b-304ad5f1abf44548a72961d3f655cf61 on tablet 304ad5f1abf44548a72961d3f655cf61 with peer 84305d92b082406380b107114e4cd61b
@ 0x7f73c136f4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f73c135a226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f73c1369c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ea1821575c std::function<>::operator()() at ??:0
@ 0x7f73be723e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f73be7246ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 08:00:55.492401 22679 ts_tablet_manager.cc:902] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Initiating tablet copy from peer 78e52cf3c8fa435bb5ca36061227ac41 (127.0.99.134:35175)
I20240503 08:00:55.492852 22679 tablet_copy_client.cc:308] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:35175
I20240503 08:00:55.493279 17946 tablet_copy_service.cc:140] P 78e52cf3c8fa435bb5ca36061227ac41: Received BeginTabletCopySession request for tablet ab988cc96fc940239e18f54ec9c9b401 from peer 84305d92b082406380b107114e4cd61b ({username='slave'} at 127.0.99.133:48107)
I20240503 08:00:55.493363 17946 tablet_copy_service.cc:161] P 78e52cf3c8fa435bb5ca36061227ac41: Beginning new tablet copy session on tablet ab988cc96fc940239e18f54ec9c9b401 from peer 84305d92b082406380b107114e4cd61b at {username='slave'} at 127.0.99.133:48107: session id = 84305d92b082406380b107114e4cd61b-ab988cc96fc940239e18f54ec9c9b401
I20240503 08:00:55.494115 17946 tablet_copy_source_session.cc:205] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f73be72603b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ea1821575c std::function<>::operator()() at ??:0
I20240503 08:00:55.494946 22679 ts_tablet_manager.cc:1875] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f73be713807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73bef516db start_thread at ??:0
@ 0x7f73bc48971f clone at ??:0
I20240503 08:00:55.506091 22711 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.506327 22711 tablet_bootstrap.cc:492] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Bootstrap complete.
I20240503 08:00:55.506609 22711 ts_tablet_manager.cc:1367] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.049s user 0.028s sys 0.000s
I20240503 08:00:55.507107 22679 ts_tablet_manager.cc:1888] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 08:00:55.507050 22711 raft_consensus.cc:348] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:55.507213 22711 raft_consensus.cc:729] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: LEARNER
I20240503 08:00:55.507323 22679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab988cc96fc940239e18f54ec9c9b401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:00:55.507311 22711 consensus_queue.cc:260] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:55.507707 22711 ts_tablet_manager.cc:1398] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:00:55.508319 17286 tablet_copy_service.cc:342] P 8eb6822290b046f88c861159aa9421ec: Request end of tablet copy session 84305d92b082406380b107114e4cd61b-00708b0b8f2f4366944b244a9ba8c29f received from {username='slave'} at 127.0.99.133:56087
I20240503 08:00:55.508396 17286 tablet_copy_service.cc:433] P 8eb6822290b046f88c861159aa9421ec: ending tablet copy session 84305d92b082406380b107114e4cd61b-00708b0b8f2f4366944b244a9ba8c29f on tablet 00708b0b8f2f4366944b244a9ba8c29f with peer 84305d92b082406380b107114e4cd61b
I20240503 08:00:55.509086 22679 tablet_copy_client.cc:783] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 0 data blocks...
I20240503 08:00:55.509227 22679 tablet_copy_client.cc:647] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: tablet copy: Starting download of 1 WAL segments...
I20240503 08:00:55.510644 22679 tablet_copy_client.cc:523] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:00:55.512042 22679 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Bootstrap starting.
I20240503 08:00:55.565351 22679 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:00:55.565718 22679 tablet_bootstrap.cc:492] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Bootstrap complete.
I20240503 08:00:55.565901 22679 ts_tablet_manager.cc:1367] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Time spent bootstrapping tablet: real 0.054s user 0.025s sys 0.004s
I20240503 08:00:55.566383 22679 raft_consensus.cc:348] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:55.566532 22679 raft_consensus.cc:729] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84305d92b082406380b107114e4cd61b, State: Initialized, Role: LEARNER
I20240503 08:00:55.566637 22679 consensus_queue.cc:260] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: true } }
I20240503 08:00:55.566938 22679 ts_tablet_manager.cc:1398] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:00:55.567469 17946 tablet_copy_service.cc:342] P 78e52cf3c8fa435bb5ca36061227ac41: Request end of tablet copy session 84305d92b082406380b107114e4cd61b-ab988cc96fc940239e18f54ec9c9b401 received from {username='slave'} at 127.0.99.133:48107
I20240503 08:00:55.567551 17946 tablet_copy_service.cc:433] P 78e52cf3c8fa435bb5ca36061227ac41: ending tablet copy session 84305d92b082406380b107114e4cd61b-ab988cc96fc940239e18f54ec9c9b401 on tablet ab988cc96fc940239e18f54ec9c9b401 with peer 84305d92b082406380b107114e4cd61b
I20240503 08:00:55.621611 17662 raft_consensus.cc:1203] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 08:00:55.709671 17862 connection.cc:535] client connection to 127.0.99.137:44583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.709686 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.99.137:44583 (error 108)
W20240503 08:00:55.709755 17994 connection.cc:535] client connection to 127.0.99.137:44583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.709818 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.709847 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.710407 17466 connection.cc:535] client connection to 127.0.99.137:44583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.710536 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.711221 17598 connection.cc:535] client connection to 127.0.99.137:44583 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.711337 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.711613 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:00:55.711687 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.712765 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.712914 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.712986 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.713492 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:55.740761 17528 raft_consensus.cc:1203] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 08:00:55.746573 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.760272 17994 connection.cc:535] client connection to 127.0.99.136:41085 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.760360 17862 connection.cc:535] client connection to 127.0.99.136:41085 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.761004 17466 connection.cc:535] client connection to 127.0.99.136:41085 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.761193 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.99.136:41085 (error 108)
W20240503 08:00:55.761773 17334 connection.cc:535] client connection to 127.0.99.136:41085 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:00:55.762336 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:55.760798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:41085 (local address 127.0.99.135:34017)
0503 08:00:55.760929 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:55.760942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 08:00:55.761028 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:00:55.761832 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":48}
W20240503 08:00:55.762352 22819 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:00:55.761460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:41085 (local address 127.0.99.131:38065)
0503 08:00:55.761825 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0503 08:00:55.761878 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":300}
W20240503 08:00:55.762531 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.762732 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.762804 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.763440 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111)
W20240503 08:00:55.764287 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.764907 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.765187 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.766185 17202 consensus_peers.cc:462] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111)
W20240503 08:00:55.782619 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.785717 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.803220 17202 consensus_peers.cc:462] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111)
W20240503 08:00:55.817373 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.818522 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.837728 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.839522 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:55.840071 17793 raft_consensus.cc:1203] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 08:00:55.841384 17793 raft_consensus.cc:1203] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 08:00:55.843523 17398 raft_consensus.cc:1203] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 08:00:55.864351 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:00:55.866039 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:55.869724 17793 raft_consensus.cc:1203] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 08:00:55.874953 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.884969 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.914412 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.947593 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:55.961261 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.010149 17793 raft_consensus.cc:1203] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 08:00:56.035197 17793 raft_consensus.cc:1203] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 08:00:56.040553 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:56.044423 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.104234 22866 raft_consensus.cc:1050] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc: attempting to promote NON_VOTER 0ce150b975ff4040bedde6319d5cf3d5 to VOTER
I20240503 08:00:56.104820 22866 consensus_queue.cc:237] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false } }
I20240503 08:00:56.107106 18057 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.107230 17662 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 LEARNER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.107362 17793 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.107731 22939 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.107823 17265 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.108156 17924 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.108461 17398 raft_consensus.cc:1261] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.108825 22522 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.109166 22522 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:00:56.110699 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.110839 21898 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.111153 21898 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.111451 21898 consensus_queue.cc:1035] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.114758 22943 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false } } }
I20240503 08:00:56.115234 17924 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false } } }
I20240503 08:00:56.117071 17793 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false } } }
I20240503 08:00:56.117190 17662 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false } } }
I20240503 08:00:56.119043 17124 catalog_manager.cc:5484] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 8 to 9, 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:56.120258 17265 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false } } }
I20240503 08:00:56.120268 17398 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false } } }
I20240503 08:00:56.124624 18057 raft_consensus.cc:2932] T 143109561ad549d69a59992c02fe23af P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0ce150b975ff4040bedde6319d5cf3d5 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } attrs { promote: false } } }
I20240503 08:00:56.199923 22415 raft_consensus.cc:1050] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER 7af8f2a481294a2cb93e5761947978cc to VOTER
I20240503 08:00:56.200381 22415 consensus_queue.cc:237] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false } }
I20240503 08:00:56.202441 18057 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.202567 19504 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:00:56.202921 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.203052 17265 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.203065 17662 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.203382 17398 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.203393 22979 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.203747 22166 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.203900 22978 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.204324 17528 raft_consensus.cc:1261] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.204903 22820 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.205082 22979 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240503 08:00:56.205838 22977 consensus_queue.cc:1035] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.207602 22977 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false } } }
I20240503 08:00:56.209539 18057 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false } } }
I20240503 08:00:56.209966 17265 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false } } }
I20240503 08:00:56.210317 17120 catalog_manager.cc:5484] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 11 to 12, 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:00:56.211436 19504 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false } } }
I20240503 08:00:56.212664 17398 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false } } }
I20240503 08:00:56.215500 17662 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false } } }
I20240503 08:00:56.216638 17528 raft_consensus.cc:2932] T e57bf40798824baf8050988874664154 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 7af8f2a481294a2cb93e5761947978cc (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { replace: true } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { promote: false } } }
I20240503 08:00:56.231707 22939 raft_consensus.cc:1050] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc: attempting to promote NON_VOTER 84305d92b082406380b107114e4cd61b to VOTER
I20240503 08:00:56.232235 22939 consensus_queue.cc:237] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } }
I20240503 08:00:56.234226 18057 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.234308 17662 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.234477 17265 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.234753 17793 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.235148 22522 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.235416 22522 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.235764 17924 raft_consensus.cc:1261] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 7af8f2a481294a2cb93e5761947978cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:00:56.236232 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.236251 22941 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 08:00:56.236352 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.237550 22941 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.237761 22522 consensus_queue.cc:1035] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.239763 22522 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.240075 17926 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.242292 17265 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.243096 17793 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.243153 17120 catalog_manager.cc:5484] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7af8f2a481294a2cb93e5761947978cc" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:56.244665 17661 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.247625 18057 raft_consensus.cc:2932] T 10e598a3a124472eab35c97689057ace P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.296983 22357 raft_consensus.cc:1050] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec: attempting to promote NON_VOTER 84305d92b082406380b107114e4cd61b to VOTER
I20240503 08:00:56.297504 22357 consensus_queue.cc:237] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } }
I20240503 08:00:56.299378 17926 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:56.299424 18057 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:56.299791 17398 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:56.300190 17661 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:56.300287 17528 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:56.300671 23029 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:56.300856 23028 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:56.300997 22358 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:56.301117 17793 raft_consensus.cc:1261] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 08:00:56.301301 22358 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:56.301563 22358 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:00:56.301710 23027 consensus_queue.cc:1035] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 08:00:56.305358 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:00:56.305570 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.306952 23026 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:56.307533 17528 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:56.308797 17661 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:56.308847 17793 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:56.309638 18057 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:56.311033 17119 catalog_manager.cc:5484] T 00708b0b8f2f4366944b244a9ba8c29f P 7af8f2a481294a2cb93e5761947978cc reported cstate change: config changed from index 9 to 10, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:56.312109 17926 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:56.312247 17398 raft_consensus.cc:2932] T 00708b0b8f2f4366944b244a9ba8c29f P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } attrs { replace: true } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false replace: true } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: true } } }
I20240503 08:00:56.368631 22352 raft_consensus.cc:1050] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298: attempting to promote NON_VOTER 84305d92b082406380b107114e4cd61b to VOTER
I20240503 08:00:56.368801 23028 raft_consensus.cc:1050] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec: attempting to promote NON_VOTER f0a940e3a6e340a69ec7ac31d26c9298 to VOTER
I20240503 08:00:56.369169 22352 consensus_queue.cc:237] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } }
I20240503 08:00:56.369268 23028 consensus_queue.cc:237] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false } }
I20240503 08:00:56.371313 18057 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.371806 17265 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.371887 17793 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.371887 17528 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.371939 17791 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.372288 23027 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.372354 17661 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.372483 22351 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.372551 23027 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 08:00:56.372648 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.372797 22351 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.373117 22351 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.373375 17926 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.373402 22351 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.373926 18057 raft_consensus.cc:1261] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer f0a940e3a6e340a69ec7ac31d26c9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.373950 22351 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.374246 22351 consensus_queue.cc:1035] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.374863 17662 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.375350 19504 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.375725 17530 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.375887 23068 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.376008 23029 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 08:00:56.376178 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.376194 17398 raft_consensus.cc:1261] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEARNER]: Refusing update from remote peer 8eb6822290b046f88c861159aa9421ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.376211 23028 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.376597 23029 consensus_queue.cc:1035] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.378768 22867 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.379482 17793 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.380867 17926 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.380924 17661 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.381578 17528 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.382407 18056 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.383481 17123 catalog_manager.cc:5484] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 reported cstate change: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:56.385221 17265 raft_consensus.cc:2932] T 304ad5f1abf44548a72961d3f655cf61 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.385763 23029 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false } } }
I20240503 08:00:56.387364 17530 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false } } }
I20240503 08:00:56.387969 18056 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false } } }
I20240503 08:00:56.388674 17793 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false } } }
I20240503 08:00:56.389206 17662 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false } } }
I20240503 08:00:56.391677 19504 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false } } }
I20240503 08:00:56.391736 17115 catalog_manager.cc:5484] T 8a03fdbe7f824674a351fd4598dc2c63 P 0ce150b975ff4040bedde6319d5cf3d5 reported cstate change: config changed from index 11 to 12, f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8eb6822290b046f88c861159aa9421ec" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false } } }
I20240503 08:00:56.394364 17398 raft_consensus.cc:2932] T 8a03fdbe7f824674a351fd4598dc2c63 P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, f0a940e3a6e340a69ec7ac31d26c9298 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } attrs { replace: true } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } attrs { promote: false } } }
W20240503 08:00:56.458770 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
I20240503 08:00:56.539826 22979 raft_consensus.cc:1050] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER 84305d92b082406380b107114e4cd61b to VOTER
I20240503 08:00:56.540400 22979 consensus_queue.cc:237] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } }
I20240503 08:00:56.542336 17793 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.542702 17398 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.542819 18056 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.542929 17265 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.542937 17530 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.543184 17662 raft_consensus.cc:1261] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:00:56.543222 22820 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.543462 22980 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.543646 22820 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.543727 22980 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.543800 22415 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:00:56.544070 22415 consensus_queue.cc:1035] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 08:00:56.544510 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.546540 22977 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.547075 17398 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P f0a940e3a6e340a69ec7ac31d26c9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.547259 18056 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.548699 17793 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.549715 17120 catalog_manager.cc:5484] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 reported cstate change: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:00:56.549819 17265 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.549821 17530 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.553340 17662 raft_consensus.cc:2932] T 70f073cc41d54638a2c51746b94a5ece P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "f0a940e3a6e340a69ec7ac31d26c9298" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43225 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "0665f4f85b984a8bba213ad479a88bad" member_type: VOTER last_known_addr { host: "127.0.99.137" port: 44583 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.554224 22820 raft_consensus.cc:1050] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41: attempting to promote NON_VOTER 84305d92b082406380b107114e4cd61b to VOTER
I20240503 08:00:56.554684 22820 consensus_queue.cc:237] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } }
I20240503 08:00:56.556795 17793 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 LEARNER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.557027 19504 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.557101 17530 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240503 08:00:56.557605 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:00:56.557760 17265 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.557746 22978 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.558028 22978 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.558081 17662 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.558282 22978 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.558513 22978 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.559011 22976 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.559103 18056 raft_consensus.cc:1261] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Refusing update from remote peer 78e52cf3c8fa435bb5ca36061227ac41: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 08:00:56.559674 22979 consensus_queue.cc:1035] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 08:00:56.561887 22976 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.562330 18056 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P d502c5c7f12a4e39aa25cd57f92c403d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.562966 17793 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.563443 17662 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 0ce150b975ff4040bedde6319d5cf3d5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.564934 17530 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 7af8f2a481294a2cb93e5761947978cc [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.565078 17265 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P 8eb6822290b046f88c861159aa9421ec [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.565552 17116 catalog_manager.cc:5484] T ab988cc96fc940239e18f54ec9c9b401 P 84305d92b082406380b107114e4cd61b reported cstate change: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e52cf3c8fa435bb5ca36061227ac41" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
I20240503 08:00:56.569283 19504 raft_consensus.cc:2932] T ab988cc96fc940239e18f54ec9c9b401 P ab18888a9724446b9ee4fb9efe811b09 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 84305d92b082406380b107114e4cd61b (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "78e52cf3c8fa435bb5ca36061227ac41" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 35175 } } peers { permanent_uuid: "0ce150b975ff4040bedde6319d5cf3d5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34863 } } peers { permanent_uuid: "d502c5c7f12a4e39aa25cd57f92c403d" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41359 } attrs { replace: true } } peers { permanent_uuid: "8eb6822290b046f88c861159aa9421ec" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33199 } } peers { permanent_uuid: "7af8f2a481294a2cb93e5761947978cc" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39647 } } peers { permanent_uuid: "ab18888a9724446b9ee4fb9efe811b09" member_type: VOTER last_known_addr { host: "127.0.99.138" port: 33637 } attrs { promote: false } } peers { permanent_uuid: "b65378a282b9443484843de0b8906df9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "84305d92b082406380b107114e4cd61b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38825 } attrs { promote: false } } }
W20240503 08:00:57.029225 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:00:57.451172 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:00:57.679383 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:57.905912 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:00:57.932562 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:57.980829 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.020879 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.057289 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.100277 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.111531 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.195935 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.220764 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.223418 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.280650 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.282212 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.289391 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.393194 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.453156 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:00:58.453970 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.519745 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.552325 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.564790 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.589306 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.666386 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.708273 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.751462 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.762156 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.779114 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.784478 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.847780 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.932678 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:00:58.969576 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:58.972527 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:59.044420 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:00:59.435895 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:00:59.978704 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:00.046579 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.280354 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.349045 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.415323 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:00.446579 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.510053 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.625819 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.691257 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.700629 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.701268 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.708077 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.715831 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:01:00.738828 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.767369 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:01:00.767948 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.800297 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:01:00.803357 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:01:00.817663 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.841177 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 08:01:00.843523 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:01:00.844178 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.884768 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.887308 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.934825 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.942202 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:00.961241 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:00.976362 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:01.027779 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.032091 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.036288 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.173243 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.299486 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.317230 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.328131 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.344287 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.472770 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:01.543807 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.579538 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.581990 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:01:01.999521 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:02.437660 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:02.545361 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:02.713972 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:02.835661 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:02.862108 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:02.886829 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.048288 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.078764 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.143777 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.159740 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.222819 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.231294 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.278820 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.300256 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.320696 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:03.327270 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.409534 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.440908 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.485402 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.523867 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.525902 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.614665 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.710927 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.750882 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.753969 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.756035 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:03.796630 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.827525 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.863081 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.926393 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:03.999462 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:04.067137 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:04.234212 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 08:01:04.290632 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:04.856874 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:04.944538 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.335381 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:05.340351 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.386788 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.394002 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.410290 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.479635 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.563843 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.573985 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.757416 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:01:05.758021 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.778442 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.792028 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:01:05.825685 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.837489 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.843763 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.883373 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:01:05.883973 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:05.925886 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:01:05.931588 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:05.959767 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:05.960361 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.008713 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.011333 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.015367 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:01:06.015920 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.098996 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.164820 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.243638 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.316192 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.383251 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.390653 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.411576 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:06.440934 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:06.446343 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.488324 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.509662 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.623067 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.633646 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 08:01:06.931021 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:07.391172 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:07.587288 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:07.814764 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:07.825368 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:07.903407 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:07.964502 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:07.969579 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.024094 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.093397 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.117270 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.130692 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.158684 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.388813 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.390197 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.394726 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.420396 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.424707 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.456784 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:08.485473 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.504195 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.529520 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.539427 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.647557 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.762753 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.775255 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.828212 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.841744 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.976227 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:08.978158 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.986994 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:08.997815 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:09.090123 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:09.267057 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 08:01:09.401417 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:09.878337 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:10.036934 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.141757 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.347364 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.367403 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.404209 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:10.421967 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.428274 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.507148 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.581351 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.630375 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.716207 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.796941 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.805783 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:01:10.840533 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:10.854786 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.897362 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:01:10.897933 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.904271 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.907941 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:01:10.908466 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:10.910526 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.027652 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.125660 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 08:01:11.142071 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 08:01:11.147997 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.164338 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:01:11.303130 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.310235 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.313185 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.346000 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.350754 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.425729 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.444336 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:11.451380 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.585014 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:01:11.641883 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.644134 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.689965 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.764950 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 08:01:11.922067 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:12.435187 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:12.636921 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:12.684193 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:12.770076 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:12.871742 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:12.927392 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:12.994669 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:13.080307 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.135407 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.138123 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.156180 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.230454 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.281244 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.320715 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.320972 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.356186 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.364974 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.406005 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.483582 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.581753 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.598863 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:13.671041 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.676913 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.721895 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.819566 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.956504 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:13.984375 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:14.005424 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:14.048600 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:14.186218 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:14.195573 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:14.219614 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:14.283314 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 08:01:14.813971 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:15.132073 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.170630 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.384011 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.428967 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.434808 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:15.472373 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.518644 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.559826 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.602602 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.735101 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.804170 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.825807 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.834219 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.839759 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.926257 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 08:01:15.939594 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240503 08:01:15.957334 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:01:15.958029 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:15.974392 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:16.008474 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.134719 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.136283 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.177127 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:01:16.207161 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.250610 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:16.251147 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.290617 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.293395 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.403661 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.403756 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:16.434808 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:01:16.517884 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.568863 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.613374 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.653234 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.746140 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.788218 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:16.850345 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:16.910667 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:16.926748 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 08:01:17.389367 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:17.610030 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:17.800742 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:17.886968 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:17.944221 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:17.973132 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:17.984231 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.015187 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.016312 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.066663 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.085595 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.253562 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.357793 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.372213 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.384852 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:18.387034 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.390038 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.459012 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.464263 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.584641 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.770120 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.824095 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.829291 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.850679 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.944697 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.952414 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:18.988502 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:18.999547 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:19.104163 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:19.168107 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:19.227378 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:19.270291 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:19.308259 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 08:01:19.357683 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:19.857952 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:19.977042 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.018402 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.241142 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:20.289263 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.511554 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.548851 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.585423 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.611454 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.641997 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.685758 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.719857 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.771039 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.775668 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.848130 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:20.958209 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:01:20.959111 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.969460 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:20.999922 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240503 08:01:21.000533 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.001464 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:01:21.046583 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.108019 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.156064 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.237473 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:01:21.256995 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.257822 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.284641 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:21.370568 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.380012 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.381913 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:21.437772 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.558380 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:01:21.584930 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.617233 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.630717 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.694519 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.761476 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.910957 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 08:01:21.958216 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:22.153798 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:22.178982 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:01:22.439252 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:22.760326 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:22.835978 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:22.910748 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:22.934427 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.024807 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.110760 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.128289 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.169659 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.183342 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.252889 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.333509 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.341886 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.409616 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:23.466289 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.496068 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.502840 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.567095 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.627429 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.651775 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.712775 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.781970 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.790973 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.802572 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:23.871095 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:23.945372 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:24.056421 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:24.073684 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:24.096472 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:24.239600 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:24.302122 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:24.345413 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:24.364723 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 08:01:24.741814 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:24.798905 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.146881 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.280629 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.294165 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:25.414211 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.471146 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.537874 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.633288 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.678052 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.714344 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.761657 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.788942 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.817803 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.821228 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:25.889988 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.913232 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.951532 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:25.982728 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.000609 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:01:26.001323 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.012975 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:01:26.030683 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:01:26.096037 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.172654 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.226673 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:26.264341 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:01:26.264921 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.293601 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:26.294327 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.314258 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.359747 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.548512 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.552400 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.556458 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.588815 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.807158 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:01:26.842561 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:26.847875 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:27.037829 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 08:01:27.179026 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:27.400786 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:01:27.439715 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:27.801302 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:27.807638 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:27.817142 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:27.852437 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:27.945231 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:27.992347 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.117652 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.196025 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.198802 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.199232 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.247785 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.321791 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.358700 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.414072 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.418130 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.439482 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:28.488018 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.589829 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.591041 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.592347 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.689862 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.695802 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.720023 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.784354 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:28.818909 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:29.004534 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:29.125584 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:29.202060 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:29.237942 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:29.304574 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:29.313338 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:29.382033 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 08:01:29.671798 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:29.747332 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:30.058583 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.256114 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.290079 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.343060 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:30.503648 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.510753 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.513187 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.618356 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.620875 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.836686 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.861218 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.868757 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.873847 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.928210 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.941154 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:30.944244 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:30.983292 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.034508 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.072865 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.113366 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:01:31.181880 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:01:31.249377 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:01:31.249466 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.250041 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.261565 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.318359 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:01:31.318939 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.391177 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:31.424389 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.428431 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 08:01:31.455551 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.558431 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.669580 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.677911 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.801259 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:31.821465 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:31.836676 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 08:01:32.038944 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:01:32.205302 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:32.354192 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:32.397046 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:32.425365 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:01:32.639981 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:32.732854 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:32.761528 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:32.810873 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:32.902383 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:32.932826 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.079828 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.229007 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.255883 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.268594 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.356369 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.439352 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.442950 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:33.453186 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.514722 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.557861 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.580737 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.779311 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.780103 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.930409 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:33.994127 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:34.017364 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:34.023118 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:34.064023 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:34.206526 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:34.243930 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:34.505715 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:34.510001 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:34.661729 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:34.661801 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:34.688122 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:34.903491 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:34.986511 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 08:01:35.047384 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.056124 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:35.057574 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.131495 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.420439 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.501919 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.540105 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.580291 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.607592 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.671522 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:35.672222 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.715925 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.767073 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.898095 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:35.909304 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.064770 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.093777 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:36.130956 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.171598 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:01:36.219964 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:01:36.220479 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.296501 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.299410 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:01:36.326325 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20240503 08:01:36.339169 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.374450 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.452935 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:36.478830 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.591481 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.620914 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.630004 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.658418 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:36.834416 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:36.939412 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:37.014555 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:37.030592 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:37.159615 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:01:37.268846 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:37.419788 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:37.505458 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 08:01:37.510843 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:37.539542 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:37.550356 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:37.652349 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:37.731281 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 08:01:37.732199 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:37.839152 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:37.926961 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:37.997268 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.136001 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.216689 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.246316 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.283778 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.382777 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.409226 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:38.416468 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.443657 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.581565 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.629974 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.664312 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.839756 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.865710 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:38.878890 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:38.945322 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:39.122004 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:39.236850 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:39.276294 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:39.291466 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:39.300932 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:39.389534 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:39.646878 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:39.828604 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:39.935205 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:39.942445 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 08:01:39.964792 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.013402 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:40.029731 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.105456 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.111555 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.228109 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.362856 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.476095 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.537037 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.553483 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.601083 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.603082 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:40.627677 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.750753 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:40.930907 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.015637 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.053018 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.082394 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.126055 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.155844 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:41.247896 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.342842 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:01:41.451022 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:01:41.461359 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:01:41.496490 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:41.509624 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:01:41.510140 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.513701 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.553946 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.566053 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:41.596141 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.677649 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:41.949488 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:42.053658 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:42.106083 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:42.250104 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:42.278012 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 08:01:42.290650 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:01:42.445825 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:42.467594 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:42.496268 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:42.542445 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:42.549479 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:42.585688 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:42.788260 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:42.858964 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:42.957219 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:42.977715 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:43.001909 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:43.044458 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.047627 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.114723 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.180099 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.207252 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.208051 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.226794 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.353291 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:43.419646 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.428258 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.433672 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.524119 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.680244 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.725265 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.804042 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:43.907999 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:43.941257 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:44.060015 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:44.114534 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:44.369388 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:44.406080 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:44.624053 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:44.624702 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:44.828931 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:44.838335 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:44.844151 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:44.921000 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 08:01:44.954761 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:44.961042 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:44.998241 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.190893 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.232743 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.341217 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.467713 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.516011 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.562175 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.567143 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:45.620904 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.728961 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.741613 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.758378 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.860828 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.901255 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.909437 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:45.921339 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:46.126042 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:46.200050 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:46.219372 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:46.363843 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:01:46.472858 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:01:46.504912 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:46.505474 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:46.517859 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:01:46.526466 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:01:46.562057 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:46.571507 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:46.571830 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:46.699771 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:47.002882 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.037154 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:47.150741 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:47.257534 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:47.336301 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.382354 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:01:47.427009 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.490100 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:47.532713 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:47.580032 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.678985 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.725359 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 08:01:47.751598 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.764822 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.790169 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.816697 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.920689 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.970391 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:47.978472 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:48.160399 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.212831 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.275748 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.295825 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.301398 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:48.372144 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.390676 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:48.400549 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.468130 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.483532 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.499641 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.803258 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.859814 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.880228 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:48.949213 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:49.069203 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:49.146333 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:49.385959 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:49.560456 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:49.712751 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:49.731443 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:49.768352 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:49.774422 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:49.776100 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:49.986214 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:50.048565 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.112779 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.127696 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.193800 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 08:01:50.318961 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.319964 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.320338 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.335428 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.365869 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.390100 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:50.617996 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.622885 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.780016 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.816583 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.826577 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.906850 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:50.958313 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:50.989706 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:51.037812 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:51.069746 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:51.125797 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:51.305585 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:51.378278 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:51.506460 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 08:01:51.525337 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:01:51.525949 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:51.656436 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:51.690430 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:01:51.723840 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:51.736204 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:51.749994 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:01:51.838565 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:51.925086 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.072221 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.120087 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:52.213025 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:52.364346 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:52.371408 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:52.398018 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:01:52.492580 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.564123 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.617650 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.678776 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.697403 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:52.875996 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.904588 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.942600 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.962307 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:52.983644 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.013778 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 08:01:53.241652 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.292575 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:53.301005 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.360211 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.381549 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.467012 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.500167 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.544564 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:53.713790 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.780879 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.785795 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.791049 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:53.893893 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:53.914999 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:54.150709 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:54.274618 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:54.303133 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:54.303640 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:54.361390 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:54.503101 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:54.503276 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:54.738507 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:54.749910 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:54.812137 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.056403 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.215644 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.229104 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:55.323904 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.334019 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:55.349512 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.365454 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.397281 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 08:01:55.539884 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.611124 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.644217 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.665117 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.676344 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:55.827003 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:55.871304 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.047659 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.054925 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.186745 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.278371 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.363924 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.400254 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.450815 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:56.488533 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.527855 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.545217 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 08:01:56.612013 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.671314 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:01:56.671888 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:56.821709 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:01:56.842444 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:01:56.853629 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:56.875423 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:56.923532 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:01:57.010864 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:57.063091 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:57.065405 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:57.163028 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:57.285959 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:57.437363 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:57.461293 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:01:57.595257 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:57.608707 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:57.664129 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 08:01:57.696471 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:57.738353 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:57.880224 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:57.895066 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:57.948801 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:57.979302 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.089669 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.095733 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.100910 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.387928 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.442212 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:58.469661 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.521279 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.556741 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.600548 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.843670 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:58.851917 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.852768 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:58.893744 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:01:58.901036 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:59.012717 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:59.019966 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:59.063912 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:59.212802 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:59.262296 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:59.317858 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:59.339433 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:59.487955 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:59.499516 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:59.600595 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:01:59.675124 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:01:59.915081 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:01:59.985623 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.065099 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.066608 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.097512 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:00.281677 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.350435 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.400775 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.496191 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.496659 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 08:02:00.518213 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.528261 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.682514 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.693675 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:00.878394 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:00.909622 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.030825 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.032370 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.072702 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.311300 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:01.343583 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.366290 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.413628 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.445328 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.494696 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.496598 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:01.574560 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:02:01.703174 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:02:01.826205 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:01.868403 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:01.877120 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:02:01.880447 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:02:01.930707 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:02:01.972182 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:01.998054 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:02.062693 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:02.145936 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:02.169198 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:02.276223 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:02.345966 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:02.501588 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:02.519641 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:02.625169 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:02.639070 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:02.786937 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 08:02:02.787504 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:02.846403 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:02.856967 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:02.857407 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 08:02:02.986886 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.150552 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.154037 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.179500 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.320613 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.343228 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.380303 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:03.559223 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.564369 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.670506 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.729801 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.834306 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.839376 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:03.849189 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:03.915122 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:04.056728 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:04.065361 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:02:04.070680 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:04.238126 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:04.262149 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:04.432379 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:04.450381 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:04.482582 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:04.523067 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:04.796705 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:04.861248 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:04.929968 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:04.963593 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:05.139933 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:05.151775 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:05.182261 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:05.225148 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:05.280838 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 08:02:05.282959 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:05.396126 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:05.519094 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:05.848641 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:05.859515 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:05.903580 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:05.922061 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:05.956976 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.002863 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:06.025300 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.091926 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.223601 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.242558 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.359769 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.434587 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.451337 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:06.481568 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.517216 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.541517 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:06.601208 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.632741 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:02:06.763679 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:02:06.765460 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:06.921814 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:02:06.933084 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:02:06.933908 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:06.993876 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:07.037676 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:02:07.038241 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:07.071278 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:07.137609 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:07.285297 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:07.314193 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:07.364027 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:07.549448 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:07.632247 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:07.662281 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 08:02:07.692570 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:07.723843 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:07.760279 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:07.858839 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:02:08.041616 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.249215 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:08.253554 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.262135 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.289125 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.552467 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.554733 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.567046 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.626773 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.653604 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:08.680941 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.761907 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:08.771606 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.786013 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:08.947427 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:09.044631 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:09.199615 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:09.270044 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:09.291913 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:09.347010 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:02:09.347623 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:09.350260 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:09.441373 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:09.466692 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:09.556898 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:09.597034 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:09.820178 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:09.840550 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:09.865216 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:09.890165 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:10.009198 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:10.132445 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:10.232064 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 08:02:10.320667 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:10.395190 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:10.449263 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:10.679206 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:10.821292 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:10.983024 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:10.992146 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:10.995458 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:10.995924 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:10.999056 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.245443 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:11.301891 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.340286 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.341015 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.385720 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.462574 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.506906 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.553488 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:11.664888 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:02:11.845329 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.883180 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.893414 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.907240 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 08:02:11.921876 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:02:11.973104 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:11.975970 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:11.993598 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:02:12.047120 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:12.067847 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:12.069996 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:12.070180 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:12.205593 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 08:02:12.299407 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:12.318776 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:12.413116 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:12.449843 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:12.462388 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:12.709807 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:12.785001 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 08:02:12.837605 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:12.926553 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:02:13.051906 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:13.056421 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.182405 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.252105 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.333590 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.337674 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.473630 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.484839 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:13.488761 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.649317 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.778193 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.825826 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:13.893824 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:13.900417 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:14.018460 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:14.044251 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:14.086310 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:14.153731 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:14.199103 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:14.218665 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:14.222604 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:14.349823 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 08:02:14.350559 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:14.366672 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:14.481179 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:14.520423 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:14.523020 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:14.596540 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:14.653941 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:14.764642 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:14.962182 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:15.058517 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:15.120621 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:15.248615 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 08:02:15.395589 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:15.516407 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:15.531265 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:15.654372 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:15.663028 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:15.672818 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:15.932629 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:15.937249 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.014029 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.014467 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:16.142096 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:16.175170 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.185009 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.214671 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.583004 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:16.646577 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:16.669382 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 08:02:16.670063 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:16.695326 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.705123 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.713582 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:16.733335 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.798086 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.837841 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:16.838551 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:16.973876 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 08:02:16.994804 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:17.000108 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240503 08:02:17.000718 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:17.043053 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 08:02:17.043601 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:17.090986 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:17.177148 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:17.310465 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 08:02:17.311122 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:17.415208 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:17.430696 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:17.506579 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:17.885522 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 08:02:17.941460 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:17.988021 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:17.999372 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:18.080446 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:18.195648 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:02:18.202834 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:18.334105 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:18.334124 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:18.374114 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:18.587846 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:18.618145 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:18.637800 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:18.691953 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:18.719233 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:18.730876 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:18.813472 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:19.081598 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:19.098399 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:19.100332 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:19.140892 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:19.147110 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:19.157987 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:19.168788 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:19.202564 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:19.296649 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:19.307449 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:19.416395 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:19.526394 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:19.649850 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:19.691125 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:19.737182 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:02:19.752382 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:19.773942 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:20.058391 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:20.161177 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:20.279912 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:20.484892 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:20.545157 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:20.554957 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:20.622870 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 08:02:20.773147 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:20.815479 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:20.825762 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:20.961573 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:20.964273 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.039000 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:21.050293 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:21.081849 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.151728 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.270310 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:21.431063 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.479394 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.504487 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:21.516932 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.521320 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.588807 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.708501 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:21.735466 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.799033 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:21.800864 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 08:02:21.801623 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:21.802232 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:21.911876 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:22.143338 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:02:22.157932 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:02:22.158540 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:22.181608 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:22.231978 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:22.314009 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240503 08:02:22.356729 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 08:02:22.396761 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:22.458307 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:22.757041 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:22.774245 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:22.776146 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:22.846757 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 08:02:22.919822 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.099444 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.134968 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.184177 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.215327 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.318363 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:23.331355 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.404919 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:23.440513 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:02:23.562681 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.619841 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:23.664090 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.737970 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.750016 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.778605 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:23.792567 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:23.875826 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:23.979328 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:24.025979 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:24.050316 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:24.137928 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:24.221417 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:24.268576 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:24.328420 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:24.446323 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:24.465247 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:24.559516 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:24.571880 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:24.894613 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:24.988343 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:25.003096 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:25.059984 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:02:25.094503 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:25.330569 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 08:02:25.357291 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:25.394104 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:25.478135 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:25.499619 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:25.568864 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:25.609613 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:25.699719 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:25.834671 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:25.863577 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:25.996912 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:26.062112 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.131526 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:26.152174 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.281152 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.289917 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.325654 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.331697 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:26.365039 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.383702 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.383791 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.408071 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:26.514614 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.550109 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:26.569034 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:26.746399 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:26.790367 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:27.013163 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240503 08:02:27.154922 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 08:02:27.156529 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:27.166414 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:02:27.180521 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:27.197129 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:27.275981 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:27.280782 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:27.394477 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:27.415586 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 08:02:27.476213 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240503 08:02:27.705555 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:27.706065 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 08:02:27.853042 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:27.857880 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:27.881893 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.019615 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.043607 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.086818 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.165511 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:28.170364 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.264101 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.359591 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:28.462680 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 08:02:28.544078 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:28.590327 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.660591 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:28.666599 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.709020 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.778919 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.780329 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.780861 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:28.789328 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.808169 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.809109 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:28.837097 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:28.993012 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:29.074162 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:29.189529 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:29.329718 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:29.359362 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:29.541301 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:29.671054 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:29.737126 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:29.812810 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:30.007273 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:30.262334 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:30.269750 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:30.282378 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 08:02:30.343798 17730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 08:02:30.414387 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:30.468091 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:30.540624 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:30.546902 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:30.679734 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:30.735064 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:30.821766 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:30.838675 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:30.848723 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.012388 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.068157 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.128724 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.186359 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.217839 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:31.236426 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:31.282480 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.295110 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.419924 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:31.428275 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:31.449863 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.470309 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.510900 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.589596 17334 consensus_peers.cc:485] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:31.800241 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:31.985427 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:32.010434 17202 consensus_peers.cc:485] T 8a03fdbe7f824674a351fd4598dc2c63 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:32.021880 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:32.156633 17598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 08:02:32.170270 17598 consensus_peers.cc:485] T 882c64fe082243fbbb3099f2bf6e33e6 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:32.175714 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 08:02:32.184383 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240503 08:02:32.365304 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:32.395920 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:32.445639 17466 consensus_peers.cc:485] T 143109561ad549d69a59992c02fe23af P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 08:02:32.482573 17862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 08:02:32.515648 17113 maintenance_manager.cc:419] P 51daab6ec19c401bbaa500647c2d47dd: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033363
I20240503 08:02:32.534149 17112 maintenance_manager.cc:643] P 51daab6ec19c401bbaa500647c2d47dd: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.013s sys 0.004s Metrics: {"bytes_written":21447,"cfile_init":1,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":874,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":491,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":31,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":115}
I20240503 08:02:32.534919 17113 maintenance_manager.cc:419] P 51daab6ec19c401bbaa500647c2d47dd: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 9118 bytes on disk
I20240503 08:02:32.535367 17112 maintenance_manager.cc:643] P 51daab6ec19c401bbaa500647c2d47dd: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240503 08:02:32.570487 17334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 08:02:32.677700 17862 consensus_peers.cc:485] T e57bf40798824baf8050988874664154 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:32.773452 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:32.843379 17994 consensus_peers.cc:485] T 08e9b83cd82f4d82ae552501be742896 P d502c5c7f12a4e39aa25cd57f92c403d -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:32.870054 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:32.992128 17862 consensus_peers.cc:485] T 70f073cc41d54638a2c51746b94a5ece P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.129204 17862 consensus_peers.cc:485] T ab988cc96fc940239e18f54ec9c9b401 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.160547 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.180078 17334 consensus_peers.cc:485] T 8500b198108649a7bcb614c93cf53a3a P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.279170 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:33.314993 17598 consensus_peers.cc:485] T ba0615c2fbc840c7bbdfda22caf45de2 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.366564 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:33.442559 17466 consensus_peers.cc:485] T a79ddd7d381c4ccbbd1a1e7d3ec8366f P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.474442 17334 consensus_peers.cc:485] T 42be6135d7654954b6886c32dfc688c5 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.527660 17994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 08:02:33.613983 17202 consensus_peers.cc:485] T 00708b0b8f2f4366944b244a9ba8c29f P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.674770 17202 consensus_peers.cc:485] T 99d8d89b8561420781b7e7a9b9c778a0 P 8eb6822290b046f88c861159aa9421ec -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.719470 17334 consensus_peers.cc:485] T ed32a1099e044448aaa0c9c9f58f74e6 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:33.728330 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.779532 17466 consensus_peers.cc:485] T 10e598a3a124472eab35c97689057ace P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.800906 17334 consensus_peers.cc:485] T 304ad5f1abf44548a72961d3f655cf61 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.865273 17862 consensus_peers.cc:485] T 15a5f02e61d24029b0a5757556085733 P 78e52cf3c8fa435bb5ca36061227ac41 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 08:02:33.911417 17334 consensus_peers.cc:462] T 3d1de048a83e45548afac96309abf4d2 P f0a940e3a6e340a69ec7ac31d26c9298 -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111)
W20240503 08:02:33.916049 17730 consensus_peers.cc:485] T bbe0edacc8ce4ba8be5e533c82ea92e0 P 84305d92b082406380b107114e4cd61b -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn't send request to peer b65378a282b9443484843de0b8906df9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41085: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:33.989753 17466 consensus_peers.cc:485] T 5154e7f64d1542979f9363a429fa469c P 7af8f2a481294a2cb93e5761947978cc -> Peer 0665f4f85b984a8bba213ad479a88bad (127.0.99.137:44583): Couldn't send request to peer 0665f4f85b984a8bba213ad479a88bad. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.137:44583: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 08:02:33.990031 17598 consensus_peers.cc:485] T ecc1f8b69268429f89cffb380e45e020 P 0ce150b975ff4040bedde6319d5cf3d5 -> Peer b65378a282b9443484843de0b8906df9 (127.0.99.136:41085): Couldn'